01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:32:04 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:32:04 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:32:04 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:32:04 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:32:04 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:32:05 AM UTC [DEBUG] Node channel sub id: 102436 01/07/2026 06:32:05 AM UTC [DEBUG] Retry channel sub id: 102437 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:32:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:34] "GET /health HTTP/1.1" 200 - 01/07/2026 06:42:34 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:34 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:42:34 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:32:04 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:32:04 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:32:04 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:32:04 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:32:04 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:32:05 AM UTC [DEBUG] Node channel sub id: 102436 01/07/2026 06:32:05 AM UTC [DEBUG] Retry channel sub id: 102437 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:32:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:34] "GET /health HTTP/1.1" 200 - 01/07/2026 06:42:34 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:34 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:42:34 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:36 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:42:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:42:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:42:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:42:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:42:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:42:37 AM UTC [DEBUG] Node channel sub id: 102438 01/07/2026 06:42:37 AM UTC [DEBUG] Retry channel sub id: 102439 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:43:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:43:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:53:06] "GET /health HTTP/1.1" 200 - 01/07/2026 06:53:07 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:07 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:53:07 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:32:04 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:32:04 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:32:04 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:32:04 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:32:04 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:32:05 AM UTC [DEBUG] Node channel sub id: 102436 01/07/2026 06:32:05 AM UTC [DEBUG] Retry channel sub id: 102437 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:32:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:34] "GET /health HTTP/1.1" 200 - 01/07/2026 06:42:34 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:34 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:42:34 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:36 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:42:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:42:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:42:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:42:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:42:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:42:37 AM UTC [DEBUG] Node channel sub id: 102438 01/07/2026 06:42:37 AM UTC [DEBUG] Retry channel sub id: 102439 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:43:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:43:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:53:06] "GET /health HTTP/1.1" 200 - 01/07/2026 06:53:07 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:07 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:53:07 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:09 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:53:09 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:53:09 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:53:09 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:53:09 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:53:09 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:53:09 AM UTC [DEBUG] Node channel sub id: 102440 01/07/2026 06:53:09 AM UTC [DEBUG] Retry channel sub id: 102441 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:53:37] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:54:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:54:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:39] "GET /health HTTP/1.1" 200 - 01/07/2026 07:03:39 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:39 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:03:39 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:32:04 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:32:04 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:32:04 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:32:04 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:32:04 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:32:05 AM UTC [DEBUG] Node channel sub id: 102436 01/07/2026 06:32:05 AM UTC [DEBUG] Retry channel sub id: 102437 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:32:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:34] "GET /health HTTP/1.1" 200 - 01/07/2026 06:42:34 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:34 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:42:34 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:36 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:42:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:42:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:42:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:42:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:42:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:42:37 AM UTC [DEBUG] Node channel sub id: 102438 01/07/2026 06:42:37 AM UTC [DEBUG] Retry channel sub id: 102439 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:43:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:43:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:53:06] "GET /health HTTP/1.1" 200 - 01/07/2026 06:53:07 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:07 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:53:07 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:09 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:53:09 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:53:09 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:53:09 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:53:09 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:53:09 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:53:09 AM UTC [DEBUG] Node channel sub id: 102440 01/07/2026 06:53:09 AM UTC [DEBUG] Retry channel sub id: 102441 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:53:37] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:54:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:54:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:39] "GET /health HTTP/1.1" 200 - 01/07/2026 07:03:39 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:39 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:03:39 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:41 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 07:03:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 07:03:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 07:03:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 07:03:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 07:03:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 07:03:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 07:03:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 07:03:42 AM UTC [DEBUG] Node channel sub id: 102442 01/07/2026 07:03:42 AM UTC [DEBUG] Retry channel sub id: 102443 01/07/2026 07:03:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:03:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 07:03:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:03:42 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 07:04:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:04:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:05:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 07:05:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:06:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:06:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:07:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:07:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:08:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:08:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:09:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:09:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:10:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:10:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:11:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:11:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:12:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:12:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:13:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:13:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:14:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:14:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:15:11] "GET /health HTTP/1.1" 200 - 01/07/2026 07:15:12 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:15:12 AM UTC [ERROR] ================================================================================ 01/07/2026 07:15:12 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:15:12 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 07:15:12 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:32:04 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:32:04 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:32:04 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:32:04 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:32:04 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:32:05 AM UTC [DEBUG] Node channel sub id: 102436 01/07/2026 06:32:05 AM UTC [DEBUG] Retry channel sub id: 102437 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:32:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:34] "GET /health HTTP/1.1" 200 - 01/07/2026 06:42:34 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:34 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:42:34 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:36 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:42:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:42:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:42:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:42:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:42:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:42:37 AM UTC [DEBUG] Node channel sub id: 102438 01/07/2026 06:42:37 AM UTC [DEBUG] Retry channel sub id: 102439 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:43:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:43:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:53:06] "GET /health HTTP/1.1" 200 - 01/07/2026 06:53:07 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:07 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:53:07 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:09 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:53:09 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:53:09 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:53:09 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:53:09 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:53:09 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:53:09 AM UTC [DEBUG] Node channel sub id: 102440 01/07/2026 06:53:09 AM UTC [DEBUG] Retry channel sub id: 102441 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:53:37] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:54:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:54:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:39] "GET /health HTTP/1.1" 200 - 01/07/2026 07:03:39 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:39 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:03:39 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:41 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 07:03:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 07:03:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 07:03:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 07:03:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 07:03:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 07:03:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 07:03:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 07:03:42 AM UTC [DEBUG] Node channel sub id: 102442 01/07/2026 07:03:42 AM UTC [DEBUG] Retry channel sub id: 102443 01/07/2026 07:03:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:03:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 07:03:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:03:42 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 07:04:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:04:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:05:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 07:05:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:06:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:06:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:07:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:07:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:08:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:08:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:09:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:09:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:10:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:10:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:11:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:11:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:12:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:12:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:13:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:13:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:14:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:14:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:15:11] "GET /health HTTP/1.1" 200 - 01/07/2026 07:15:12 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:15:12 AM UTC [ERROR] ================================================================================ 01/07/2026 07:15:12 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:15:12 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 07:15:12 AM UTC [ERROR] ================================================================================ 01/07/2026 07:15:14 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 07:15:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 07:15:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 07:15:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 07:15:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 07:15:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 07:15:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 07:15:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 07:15:14 AM UTC [DEBUG] Node channel sub id: 102444 01/07/2026 07:15:14 AM UTC [DEBUG] Retry channel sub id: 102445 01/07/2026 07:15:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:15:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 07:15:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:15:14 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 07:15:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:16:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:16:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:17:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:17:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:18:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:18:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:19:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:19:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:20:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:20:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:21:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:21:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:22:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:22:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:23:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:23:43] "GET /health HTTP/1.1" 200 - 01/07/2026 07:23:48 AM UTC [DEBUG] Job node created: 695e0a04be553029bb40be40. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: fe005dbe3a78403ea9f93c87cf618f9a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/fe005dbe3a78403ea9f93c87cf618f9a.json 01/07/2026 07:23:48 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/fe005dbe3a78403ea9f93c87cf618f9a.json 01/07/2026 07:23:48 AM UTC [INFO] 695e0a04be553029bb40be40 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 07:23:48 AM UTC [DEBUG] Job node created: 695e0a04be553029bb40be44. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 34210aa7fa1d4cb385407a646d781f2d.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/34210aa7fa1d4cb385407a646d781f2d.json 01/07/2026 07:23:48 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/34210aa7fa1d4cb385407a646d781f2d.json 01/07/2026 07:23:49 AM UTC [INFO] 695e0a04be553029bb40be44 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 07:24:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:24:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:25:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:25:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 07:26:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:26:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:27:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:27:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:28:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:28:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:29:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:29:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:30:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:30:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:31:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:31:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:32:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:32:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:33:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:33:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:34:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:34:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:35:15] "GET /health HTTP/1.1" 200 - 01/07/2026 07:35:44 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 626s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:35:44 AM UTC [ERROR] ================================================================================ 01/07/2026 07:35:44 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:35:44 AM UTC [ERROR] Thread: scheduler-retry, stuck for 626s 01/07/2026 07:35:44 AM UTC [ERROR] ================================================================================ 01/07/2026 12:22:16 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:22:17 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:22:17 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:17 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:22:17 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:22:18 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:22:18 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:22:18 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:22:18 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:22:18 AM UTC [DEBUG] Node channel sub id: 102353 01/07/2026 12:22:18 AM UTC [DEBUG] Retry channel sub id: 102354 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:22:18 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:22:18 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:22:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:23:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:23:40] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102353 01/07/2026 12:23:45 AM UTC [INFO] Unsubscribing: 102354 01/07/2026 12:23:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:23:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:23:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:23:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:23:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:23:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:23:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:23:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:23:48 AM UTC [DEBUG] Node channel sub id: 102360 01/07/2026 12:23:48 AM UTC [DEBUG] Retry channel sub id: 102361 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:23:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:23:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:24:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:24:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:25:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:25:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:26:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:27:46] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102360 01/07/2026 12:28:02 AM UTC [INFO] Unsubscribing: 102361 01/07/2026 12:28:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:28:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:28:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:28:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:28:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:28:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:28:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:28:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:28:05 AM UTC [DEBUG] Node channel sub id: 102362 01/07/2026 12:28:05 AM UTC [DEBUG] Retry channel sub id: 102363 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:28:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:28:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:28:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:29:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:30:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:31:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:32:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:33:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:34:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:35:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:36:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:37:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:38:34] "GET /health HTTP/1.1" 200 - 01/07/2026 12:38:35 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:35 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 12:38:35 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 12:38:35 AM UTC [ERROR] ================================================================================ 01/07/2026 12:38:37 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:38:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:38:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:38:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:38:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:38:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:38:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:38:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:38:37 AM UTC [DEBUG] Node channel sub id: 102364 01/07/2026 12:38:37 AM UTC [DEBUG] Retry channel sub id: 102365 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:38:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:38:37 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:39:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:39:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:40:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:41:06] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:41:36] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102364 01/07/2026 12:41:40 AM UTC [INFO] Unsubscribing: 102365 01/07/2026 12:41:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:42 AM UTC [DEBUG] Node channel sub id: 102366 01/07/2026 12:41:42 AM UTC [DEBUG] Retry channel sub id: 102367 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:42 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102366 01/07/2026 12:41:51 AM UTC [INFO] Unsubscribing: 102367 01/07/2026 12:41:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:41:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:41:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:41:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:41:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:41:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:41:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:41:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:41:54 AM UTC [DEBUG] Node channel sub id: 102368 01/07/2026 12:41:54 AM UTC [DEBUG] Retry channel sub id: 102369 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:41:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:41:54 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:42:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102368 01/07/2026 12:42:48 AM UTC [INFO] Unsubscribing: 102369 01/07/2026 12:42:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:42:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:42:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:42:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:42:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:42:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:42:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:42:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:42:51 AM UTC [DEBUG] Node channel sub id: 102370 01/07/2026 12:42:51 AM UTC [DEBUG] Retry channel sub id: 102371 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:42:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:42:51 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102370 01/07/2026 12:43:04 AM UTC [INFO] Unsubscribing: 102371 01/07/2026 12:43:06 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:43:06 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:43:06 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:43:06 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:43:06 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:43:06 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:43:06 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:43:06 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:43:06 AM UTC [DEBUG] Node channel sub id: 102372 01/07/2026 12:43:06 AM UTC [DEBUG] Retry channel sub id: 102373 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:43:06 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:43:06 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:43:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:44:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:45:04] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102372 01/07/2026 12:45:11 AM UTC [INFO] Unsubscribing: 102373 01/07/2026 12:45:13 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:45:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:45:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:45:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:45:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:45:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:45:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:45:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:45:14 AM UTC [DEBUG] Node channel sub id: 102374 01/07/2026 12:45:14 AM UTC [DEBUG] Retry channel sub id: 102375 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:45:14 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:45:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:46:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:47:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:48:12] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:48:38 AM UTC [INFO] Unsubscribing: 102374 01/07/2026 12:48:39 AM UTC [INFO] Unsubscribing: 102375 01/07/2026 12:48:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:48:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:48:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:48:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:48:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:48:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:48:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:48:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:48:43 AM UTC [DEBUG] Node channel sub id: 102376 01/07/2026 12:48:43 AM UTC [DEBUG] Retry channel sub id: 102377 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:48:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:48:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:49:10] "GET /health HTTP/1.1" 200 - 01/07/2026 12:49:22 AM UTC [DEBUG] Job node created: 695dad92be553029bb40853e. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 235943ded6184caea82386acfb60003b.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/235943ded6184caea82386acfb60003b.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad92be553029bb40853e pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 12:49:23 AM UTC [DEBUG] Job node created: 695dad93be553029bb408543. Parent: 695da36f2317100ab3f215fd Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: dda90d02d446476d9b949abc07c821a5.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/dda90d02d446476d9b949abc07c821a5.json 01/07/2026 12:49:23 AM UTC [INFO] 695dad93be553029bb408543 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 00:49:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:50:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:51:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102376 01/07/2026 12:51:21 AM UTC [INFO] Unsubscribing: 102377 01/07/2026 12:51:23 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:51:24 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:51:24 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:51:24 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:51:24 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:51:24 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:51:24 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:51:24 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:51:24 AM UTC [DEBUG] Node channel sub id: 102378 01/07/2026 12:51:24 AM UTC [DEBUG] Retry channel sub id: 102379 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:51:24 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:51:24 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 00:51:52] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:52:22] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102378 01/07/2026 12:52:40 AM UTC [INFO] Unsubscribing: 102379 01/07/2026 12:52:42 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 12:52:43 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 12:52:43 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 12:52:43 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 12:52:43 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 12:52:43 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 12:52:43 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 12:52:43 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 12:52:43 AM UTC [DEBUG] Node channel sub id: 102380 01/07/2026 12:52:43 AM UTC [DEBUG] Retry channel sub id: 102381 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 12:52:43 AM UTC [INFO] Listening for available checkout events 01/07/2026 12:52:43 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 00:53:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:53:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:54:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:55:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:56:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:57:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:58:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 00:59:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:00:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:01:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:02:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:03:12] "GET /health HTTP/1.1" 200 - 01/07/2026 01:03:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:03:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:03:13 AM UTC [ERROR] ================================================================================ 01/07/2026 01:03:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:03:16 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:03:16 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:03:16 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:03:16 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:03:16 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:03:16 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:03:16 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:03:16 AM UTC [DEBUG] Node channel sub id: 102382 01/07/2026 01:03:16 AM UTC [DEBUG] Retry channel sub id: 102383 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:03:16 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:03:16 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:03:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:04:14] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:04:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:05:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:06:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:07:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102382 01/07/2026 01:07:44 AM UTC [INFO] Unsubscribing: 102383 01/07/2026 01:07:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:07:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:07:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:07:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:07:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:07:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:07:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:07:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:07:47 AM UTC [DEBUG] Node channel sub id: 102384 01/07/2026 01:07:47 AM UTC [DEBUG] Retry channel sub id: 102385 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:07:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:07:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:08:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:08:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:09:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:09:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:10:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:11:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:12:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:13:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:14:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:15:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:16:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:17:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:18:46] "GET /health HTTP/1.1" 200 - 01/07/2026 01:18:47 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 600s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:47 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:18:47 AM UTC [ERROR] Thread: scheduler-retry, stuck for 600s 01/07/2026 01:18:47 AM UTC [ERROR] ================================================================================ 01/07/2026 01:18:49 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:18:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:18:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:18:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:18:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:18:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:18:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:18:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:18:49 AM UTC [DEBUG] Node channel sub id: 102386 01/07/2026 01:18:49 AM UTC [DEBUG] Retry channel sub id: 102387 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:18:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:18:49 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:19:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:19:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:20:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:21:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:22:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:23:18] "GET /health HTTP/1.1" 200 - 01/07/2026 01:23:37 AM UTC [DEBUG] Job node created: 695db599be553029bb408c67. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 78ec835cbf8d4a74990d81c42514425a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/78ec835cbf8d4a74990d81c42514425a.json 01/07/2026 01:23:38 AM UTC [INFO] 695db599be553029bb408c67 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 01:23:38 AM UTC [DEBUG] Job node created: 695db59abe553029bb408c6a. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 031ecb494a0746b4874989c33c789ffa.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/031ecb494a0746b4874989c33c789ffa.json 01/07/2026 01:23:38 AM UTC [INFO] 695db59abe553029bb408c6a pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 01:23:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:24:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:25:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:26:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:27:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:28:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:29:19] "GET /health HTTP/1.1" 200 - 01/07/2026 01:29:19 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:19 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:29:19 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 01:29:19 AM UTC [ERROR] ================================================================================ 01/07/2026 01:29:21 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:29:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:29:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:29:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:29:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:29:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:29:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:29:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:29:22 AM UTC [DEBUG] Node channel sub id: 102388 01/07/2026 01:29:22 AM UTC [DEBUG] Retry channel sub id: 102389 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:29:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:29:22 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:29:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:30:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:31:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:32:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:33:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:34:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:36:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:36:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:37:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102388 01/07/2026 01:37:46 AM UTC [INFO] Unsubscribing: 102389 01/07/2026 01:37:48 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:37:49 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:37:49 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:37:49 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:37:49 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:37:49 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:37:49 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:37:49 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:37:49 AM UTC [DEBUG] Node channel sub id: 102390 01/07/2026 01:37:49 AM UTC [DEBUG] Retry channel sub id: 102391 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:37:49 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:38:17] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:38:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:39:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:40:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:41:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:42:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:43:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:44:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:45:48] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102390 01/07/2026 01:45:48 AM UTC [INFO] Unsubscribing: 102391 01/07/2026 01:45:51 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:45:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:45:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:45:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:45:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:45:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:45:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:45:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:45:51 AM UTC [DEBUG] Node channel sub id: 102392 01/07/2026 01:45:51 AM UTC [DEBUG] Retry channel sub id: 102393 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:45:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:45:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:46:49] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 01:47:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:47:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:48:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:53:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:54:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:55:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:56:51] "GET /health HTTP/1.1" 200 - 01/07/2026 01:56:51 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-node' stuck for 606s (timeout: 600s). Forcing immediate exit! 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:51 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 01:56:51 AM UTC [ERROR] Thread: scheduler-node, stuck for 606s 01/07/2026 01:56:51 AM UTC [ERROR] ================================================================================ 01/07/2026 01:56:53 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 01:56:54 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 01:56:54 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 01:56:54 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 01:56:54 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 01:56:54 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 01:56:54 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 01:56:54 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 01:56:54 AM UTC [DEBUG] Node channel sub id: 102394 01/07/2026 01:56:54 AM UTC [DEBUG] Retry channel sub id: 102395 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 01:56:54 AM UTC [INFO] Listening for available checkout events 01/07/2026 01:56:54 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 01:57:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:57:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 01:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:03:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:04:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:05:53] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:07:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:09:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:10:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:11:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:15:24] "GET /health HTTP/1.1" 200 - 01/07/2026 02:15:54 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:54 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:15:54 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 02:15:54 AM UTC [ERROR] ================================================================================ 01/07/2026 02:15:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:15:56 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:15:56 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:15:56 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:15:56 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:15:56 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:15:56 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:15:56 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:15:56 AM UTC [DEBUG] Node channel sub id: 102396 01/07/2026 02:15:56 AM UTC [DEBUG] Retry channel sub id: 102397 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:15:56 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:15:56 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:16:24] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:16:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:17:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:18:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:21:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:22:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:23:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:24:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:25:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:26:25] "GET /health HTTP/1.1" 200 - 01/07/2026 02:26:26 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:26 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:26:26 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 02:26:26 AM UTC [ERROR] ================================================================================ 01/07/2026 02:26:28 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:26:28 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:26:28 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:26:28 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:26:28 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:26:28 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:26:28 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:26:28 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:26:28 AM UTC [DEBUG] Node channel sub id: 102398 01/07/2026 02:26:28 AM UTC [DEBUG] Retry channel sub id: 102399 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:26:28 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:26:28 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:34:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:37:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:39:28] "GET /health HTTP/1.1" 200 - 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102398 Exception in thread scheduler-node: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 01/07/2026 02:39:48 AM UTC [INFO] Unsubscribing: 102399 01/07/2026 02:39:50 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:39:51 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:39:51 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:39:51 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:39:51 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:39:51 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:39:51 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:39:51 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:39:51 AM UTC [DEBUG] Node channel sub id: 102400 01/07/2026 02:39:51 AM UTC [DEBUG] Retry channel sub id: 102401 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:39:51 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:39:51 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:40:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:40:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:41:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:42:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:43:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:44:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:44:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:45:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:46:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:47:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:48:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:49:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:50:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:51:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:52:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:53:51] "GET /health HTTP/1.1" 200 - 01/07/2026 02:54:21 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 619s (timeout: 600s). Forcing immediate exit! 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:21 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 02:54:21 AM UTC [ERROR] Thread: scheduler-retry, stuck for 619s 01/07/2026 02:54:21 AM UTC [ERROR] ================================================================================ 01/07/2026 02:54:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 02:54:23 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 02:54:23 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 02:54:23 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 02:54:23 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 02:54:23 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 02:54:23 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 02:54:23 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 02:54:23 AM UTC [DEBUG] Node channel sub id: 102402 01/07/2026 02:54:23 AM UTC [DEBUG] Retry channel sub id: 102403 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 02:54:23 AM UTC [INFO] Listening for available checkout events 01/07/2026 02:54:23 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 02:54:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:55:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:56:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:57:21] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 02:57:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:58:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 02:59:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:00:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:01:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:02:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:03:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:04:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:05:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:06:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:07:53] "GET /health HTTP/1.1" 200 - 01/07/2026 03:07:53 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 621s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:53 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:07:53 AM UTC [ERROR] Thread: scheduler-retry, stuck for 621s 01/07/2026 03:07:53 AM UTC [ERROR] ================================================================================ 01/07/2026 03:07:55 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:07:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:07:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:07:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:07:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:07:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:07:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:07:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:07:55 AM UTC [DEBUG] Node channel sub id: 102404 01/07/2026 03:07:55 AM UTC [DEBUG] Retry channel sub id: 102405 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:07:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:07:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:08:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:08:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:09:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:10:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:11:54] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:12:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:12:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:13:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:14:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:15:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:16:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:17:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:18:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:19:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:20:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:21:55] "GET /health HTTP/1.1" 200 - 01/07/2026 03:21:55 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:55 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:21:55 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 03:21:55 AM UTC [ERROR] ================================================================================ 01/07/2026 03:21:57 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:21:58 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:21:58 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:21:58 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:21:58 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:21:58 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:21:58 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:21:58 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:21:58 AM UTC [DEBUG] Node channel sub id: 102406 01/07/2026 03:21:58 AM UTC [DEBUG] Retry channel sub id: 102407 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:21:58 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:21:58 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:22:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:22:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:23:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:24:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:25:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:26:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:27:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:28:27] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:28:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:29:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:30:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:31:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:32:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:33:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:34:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:35:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:36:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:37:58] "GET /health HTTP/1.1" 200 - 01/07/2026 03:38:28 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 611s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:28 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:38:28 AM UTC [ERROR] Thread: scheduler-retry, stuck for 611s 01/07/2026 03:38:28 AM UTC [ERROR] ================================================================================ 01/07/2026 03:38:30 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:38:30 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:38:30 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:38:30 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:38:30 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:38:30 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:38:30 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:38:30 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:38:30 AM UTC [DEBUG] Node channel sub id: 102408 01/07/2026 03:38:30 AM UTC [DEBUG] Retry channel sub id: 102409 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:38:30 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 03:38:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:39:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:40:28] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:40:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:41:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:42:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:43:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:44:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:45:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:46:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:47:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:48:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:49:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:50:30] "GET /health HTTP/1.1" 200 - 01/07/2026 03:50:30 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 613s (timeout: 600s). Forcing immediate exit! 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:30 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 03:50:30 AM UTC [ERROR] Thread: scheduler-retry, stuck for 613s 01/07/2026 03:50:30 AM UTC [ERROR] ================================================================================ 01/07/2026 03:50:32 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 03:50:33 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 03:50:33 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 03:50:33 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 03:50:33 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 03:50:33 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 03:50:33 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 03:50:33 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 03:50:33 AM UTC [DEBUG] Node channel sub id: 102410 01/07/2026 03:50:33 AM UTC [DEBUG] Retry channel sub id: 102411 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 03:50:33 AM UTC [INFO] Listening for available checkout events 01/07/2026 03:50:33 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 03:51:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:51:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:52:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:53:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:54:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:55:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:56:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:57:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:58:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 03:59:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:00:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:01:02] "GET /health HTTP/1.1" 200 - 01/07/2026 04:01:03 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:03 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:01:03 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 04:01:03 AM UTC [ERROR] ================================================================================ 01/07/2026 04:01:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:01:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:01:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:01:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:01:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:01:05 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:01:05 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:01:05 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:01:05 AM UTC [DEBUG] Node channel sub id: 102412 01/07/2026 04:01:05 AM UTC [DEBUG] Retry channel sub id: 102413 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:01:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:01:05 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:01:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:02:33] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:03:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:03:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:04:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:05:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:06:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:07:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:08:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:09:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:10:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:11:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:12:04] "GET /health HTTP/1.1" 200 - 01/07/2026 04:12:05 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 603s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:05 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:12:05 AM UTC [ERROR] Thread: scheduler-retry, stuck for 603s 01/07/2026 04:12:05 AM UTC [ERROR] ================================================================================ 01/07/2026 04:12:07 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:12:08 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:12:08 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:12:08 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:12:08 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:12:08 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:12:08 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:12:08 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:12:08 AM UTC [DEBUG] Node channel sub id: 102414 01/07/2026 04:12:08 AM UTC [DEBUG] Retry channel sub id: 102415 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:12:08 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:12:08 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:12:35] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:13:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:13:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:14:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:15:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:16:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:17:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:18:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:19:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:20:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:21:37] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:22:37] "GET /health HTTP/1.1" 200 - 01/07/2026 04:22:38 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 605s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:38 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:22:38 AM UTC [ERROR] Thread: scheduler-retry, stuck for 605s 01/07/2026 04:22:38 AM UTC [ERROR] ================================================================================ 01/07/2026 04:22:39 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:22:40 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:22:40 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:22:40 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:22:40 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:22:40 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:22:40 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:22:40 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:22:40 AM UTC [DEBUG] Node channel sub id: 102416 01/07/2026 04:22:40 AM UTC [DEBUG] Retry channel sub id: 102417 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:22:40 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:22:40 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:23:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:23:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:24:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:25:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:26:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:27:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:28:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:29:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:30:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:31:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:32:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:33:10] "GET /health HTTP/1.1" 200 - 01/07/2026 04:33:10 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 608s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:10 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:33:10 AM UTC [ERROR] Thread: scheduler-retry, stuck for 608s 01/07/2026 04:33:10 AM UTC [ERROR] ================================================================================ 01/07/2026 04:33:12 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:33:13 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:33:13 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:33:13 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:33:13 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:33:13 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:33:13 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:33:13 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:33:13 AM UTC [DEBUG] Node channel sub id: 102418 01/07/2026 04:33:13 AM UTC [DEBUG] Retry channel sub id: 102419 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:33:13 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:33:13 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:33:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:34:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:35:11] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:35:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:36:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:37:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:38:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:39:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:40:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:41:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:42:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:43:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:44:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:45:12] "GET /health HTTP/1.1" 200 - 01/07/2026 04:45:13 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 610s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:13 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:45:13 AM UTC [ERROR] Thread: scheduler-retry, stuck for 610s 01/07/2026 04:45:13 AM UTC [ERROR] ================================================================================ 01/07/2026 04:45:15 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:45:15 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:45:15 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:45:15 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:45:15 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:45:15 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:45:15 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:45:15 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:45:15 AM UTC [DEBUG] Node channel sub id: 102420 01/07/2026 04:45:15 AM UTC [DEBUG] Retry channel sub id: 102421 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:45:15 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:45:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:46:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:47:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:48:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:49:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:50:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:51:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:52:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:53:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:54:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:55:45] "GET /health HTTP/1.1" 200 - 01/07/2026 04:55:45 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:45 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 04:55:45 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 04:55:45 AM UTC [ERROR] ================================================================================ 01/07/2026 04:55:47 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 04:55:48 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 04:55:48 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 04:55:48 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 04:55:48 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 04:55:48 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 04:55:48 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 04:55:48 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 04:55:48 AM UTC [DEBUG] Node channel sub id: 102422 01/07/2026 04:55:48 AM UTC [DEBUG] Retry channel sub id: 102423 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 04:55:48 AM UTC [INFO] Listening for available checkout events 01/07/2026 04:55:48 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 04:56:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:56:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:57:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:58:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 04:59:16] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 04:59:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:00:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:01:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:16] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:02:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:03:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:04:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:05:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:06:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:07:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:17] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:08:47] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:09:17] "GET /health HTTP/1.1" 200 - 01/07/2026 05:09:18 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 615s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:18 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:09:18 AM UTC [ERROR] Thread: scheduler-retry, stuck for 615s 01/07/2026 05:09:18 AM UTC [ERROR] ================================================================================ 01/07/2026 05:09:19 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:09:20 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:09:20 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:09:20 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:09:20 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:09:20 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:09:20 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:09:20 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:09:20 AM UTC [DEBUG] Node channel sub id: 102424 01/07/2026 05:09:20 AM UTC [DEBUG] Retry channel sub id: 102425 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:09:20 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:09:20 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:09:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:10:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:11:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:12:48] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:18] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:13:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:14:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:15:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:16:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:17:19] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:17:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:18:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:19:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:19] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:20:49] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:21:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:22:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:23:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:24:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:25:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:26:50] "GET /health HTTP/1.1" 200 - 01/07/2026 05:27:20 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 618s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:20 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:27:20 AM UTC [ERROR] Thread: scheduler-retry, stuck for 618s 01/07/2026 05:27:20 AM UTC [ERROR] ================================================================================ 01/07/2026 05:27:22 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:27:22 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:27:22 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:27:22 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:27:22 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:27:22 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:27:22 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:27:22 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:27:22 AM UTC [DEBUG] Node channel sub id: 102426 01/07/2026 05:27:22 AM UTC [DEBUG] Retry channel sub id: 102427 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:27:22 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:27:50] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:20] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:28:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:29:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:30:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:31:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:32:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:33:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:34:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:21] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:35:51] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:36:52] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:22] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:37:52] "GET /health HTTP/1.1" 200 - 01/07/2026 05:37:52 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:52 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:37:52 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 05:37:52 AM UTC [ERROR] ================================================================================ 01/07/2026 05:37:54 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:37:55 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:37:55 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:37:55 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:37:55 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:37:55 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:37:55 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:37:55 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:37:55 AM UTC [DEBUG] Node channel sub id: 102428 01/07/2026 05:37:55 AM UTC [DEBUG] Retry channel sub id: 102429 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:37:55 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:37:55 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:38:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:38:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:39:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:40:23] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:40:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:41:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:42:53] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:23] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:43:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:44:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:45:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:46:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:47:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:48:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:49:54] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:50:24] "GET /health HTTP/1.1" 200 - 01/07/2026 05:50:25 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 607s (timeout: 600s). Forcing immediate exit! 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:25 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 05:50:25 AM UTC [ERROR] Thread: scheduler-retry, stuck for 607s 01/07/2026 05:50:25 AM UTC [ERROR] ================================================================================ 01/07/2026 05:50:27 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 05:50:27 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 05:50:27 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 05:50:27 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 05:50:27 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 05:50:27 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 05:50:27 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 05:50:27 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 05:50:27 AM UTC [DEBUG] Node channel sub id: 102430 01/07/2026 05:50:27 AM UTC [DEBUG] Retry channel sub id: 102431 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 05:50:27 AM UTC [INFO] Listening for available checkout events 01/07/2026 05:50:27 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 05:50:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:51:25] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 05:51:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:52:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:53:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:54:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:55:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:56:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:57:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:58:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 05:59:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:00:57] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:27] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:01:57] "GET /health HTTP/1.1" 200 - 01/07/2026 06:01:57 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 625s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:57 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:01:57 AM UTC [ERROR] Thread: scheduler-retry, stuck for 625s 01/07/2026 06:01:57 AM UTC [ERROR] ================================================================================ 01/07/2026 06:01:59 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:02:00 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:02:00 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:02:00 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:02:00 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:02:00 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:02:00 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:02:00 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:02:00 AM UTC [DEBUG] Node channel sub id: 102432 01/07/2026 06:02:00 AM UTC [DEBUG] Retry channel sub id: 102433 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:02:00 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:02:00 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:02:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:02:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:03:58] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:04:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:04:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:05:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:06:58] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:28] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:07:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:08:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:09:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:10:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:11:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:12:59] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:29] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:13:59] "GET /health HTTP/1.1" 200 - 01/07/2026 06:14:00 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 627s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:00 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:14:00 AM UTC [ERROR] Thread: scheduler-retry, stuck for 627s 01/07/2026 06:14:00 AM UTC [ERROR] ================================================================================ 01/07/2026 06:14:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:14:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:14:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:14:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:14:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:14:02 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:14:02 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:14:02 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:14:02 AM UTC [DEBUG] Node channel sub id: 102434 01/07/2026 06:14:02 AM UTC [DEBUG] Retry channel sub id: 102435 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:14:02 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:14:02 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:14:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:15:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:16:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:00] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:17:30] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:18:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:19:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:20:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:21:31] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:22:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:22:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:23:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:24:31] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:01] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:25:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:26:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:27:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:28:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:29:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:30:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:31:32] "GET /health HTTP/1.1" 200 - 01/07/2026 06:32:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 629s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:32:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 629s 01/07/2026 06:32:02 AM UTC [ERROR] ================================================================================ 01/07/2026 06:32:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:32:04 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:32:04 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:32:04 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:32:04 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:32:04 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:32:04 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:32:04 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:32:05 AM UTC [DEBUG] Node channel sub id: 102436 01/07/2026 06:32:05 AM UTC [DEBUG] Retry channel sub id: 102437 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:32:05 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:32:05 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:32:32] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:02] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:33:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:34:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:35:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:36:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:37:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:38:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:39:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:40:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:41:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:42:34] "GET /health HTTP/1.1" 200 - 01/07/2026 06:42:34 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:34 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:42:34 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:42:34 AM UTC [ERROR] ================================================================================ 01/07/2026 06:42:36 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:42:37 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:42:37 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:42:37 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:42:37 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:42:37 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:42:37 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:42:37 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:42:37 AM UTC [DEBUG] Node channel sub id: 102438 01/07/2026 06:42:37 AM UTC [DEBUG] Retry channel sub id: 102439 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:42:37 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:42:37 AM UTC [INFO] Press Ctrl-C to stop. Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:43:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:43:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:44:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:45:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:46:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:47:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:48:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:49:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:50:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:51:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:06] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:52:36] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:53:06] "GET /health HTTP/1.1" 200 - 01/07/2026 06:53:07 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:07 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 06:53:07 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 06:53:07 AM UTC [ERROR] ================================================================================ 01/07/2026 06:53:09 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 06:53:09 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 06:53:09 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 06:53:09 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 06:53:09 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 06:53:09 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 06:53:09 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 06:53:09 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 06:53:09 AM UTC [DEBUG] Node channel sub id: 102440 01/07/2026 06:53:09 AM UTC [DEBUG] Retry channel sub id: 102441 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 06:53:09 AM UTC [INFO] Listening for available checkout events 01/07/2026 06:53:09 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 06:53:37] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 06:54:07] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:54:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:55:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:56:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:57:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:58:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 06:59:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:00:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:08] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:01:38] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:02:39] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:09] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:03:39] "GET /health HTTP/1.1" 200 - 01/07/2026 07:03:39 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 630s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:39 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:03:39 AM UTC [ERROR] Thread: scheduler-retry, stuck for 630s 01/07/2026 07:03:39 AM UTC [ERROR] ================================================================================ 01/07/2026 07:03:41 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 07:03:42 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 07:03:42 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 07:03:42 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:03:42 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 07:03:42 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 07:03:42 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 07:03:42 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 07:03:42 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 07:03:42 AM UTC [DEBUG] Node channel sub id: 102442 01/07/2026 07:03:42 AM UTC [DEBUG] Retry channel sub id: 102443 01/07/2026 07:03:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:03:42 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 07:03:42 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:03:42 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 07:04:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:04:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:05:10] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 07:05:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:06:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:06:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:07:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:07:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:08:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:08:40] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:09:10] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:09:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:10:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:10:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:11:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:11:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:12:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:12:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:13:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:13:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:14:11] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:14:41] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:15:11] "GET /health HTTP/1.1" 200 - 01/07/2026 07:15:12 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 609s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:15:12 AM UTC [ERROR] ================================================================================ 01/07/2026 07:15:12 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:15:12 AM UTC [ERROR] Thread: scheduler-retry, stuck for 609s 01/07/2026 07:15:12 AM UTC [ERROR] ================================================================================ 01/07/2026 07:15:14 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 07:15:14 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 07:15:14 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 07:15:14 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:15:14 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 07:15:14 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 07:15:14 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 07:15:14 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 07:15:14 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 07:15:14 AM UTC [DEBUG] Node channel sub id: 102444 01/07/2026 07:15:14 AM UTC [DEBUG] Retry channel sub id: 102445 01/07/2026 07:15:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:15:14 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 07:15:14 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:15:14 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 07:15:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:16:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:16:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:17:12] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:17:42] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:18:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:18:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:19:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:19:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:20:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:20:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:21:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:21:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:22:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:22:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:23:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:23:43] "GET /health HTTP/1.1" 200 - 01/07/2026 07:23:48 AM UTC [DEBUG] Job node created: 695e0a04be553029bb40be40. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: fe005dbe3a78403ea9f93c87cf618f9a.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/fe005dbe3a78403ea9f93c87cf618f9a.json 01/07/2026 07:23:48 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/fe005dbe3a78403ea9f93c87cf618f9a.json 01/07/2026 07:23:48 AM UTC [INFO] 695e0a04be553029bb40be40 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/07/2026 07:23:48 AM UTC [DEBUG] Job node created: 695e0a04be553029bb40be44. Parent: 695da848be553029bb408424 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260107 name: 34210aa7fa1d4cb385407a646d781f2d.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/34210aa7fa1d4cb385407a646d781f2d.json 01/07/2026 07:23:48 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260107/34210aa7fa1d4cb385407a646d781f2d.json 01/07/2026 07:23:49 AM UTC [INFO] 695e0a04be553029bb40be44 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 127.0.0.1 - - [07/Jan/2026 07:24:13] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:24:43] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:25:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:25:44] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 07:26:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:26:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:27:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:27:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:28:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:28:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:29:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:29:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:30:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:30:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:31:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:31:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:32:14] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:32:44] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:33:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:33:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:34:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:34:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:35:15] "GET /health HTTP/1.1" 200 - 01/07/2026 07:35:44 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 626s (timeout: 600s). Forcing immediate exit! 01/07/2026 07:35:44 AM UTC [ERROR] ================================================================================ 01/07/2026 07:35:44 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/07/2026 07:35:44 AM UTC [ERROR] Thread: scheduler-retry, stuck for 626s 01/07/2026 07:35:44 AM UTC [ERROR] ================================================================================ 01/07/2026 07:35:46 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/07/2026 07:35:47 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:35:47 AM UTC [INFO] Runtime types from KContext: [] 01/07/2026 07:35:47 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:35:47 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:35:47 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/07/2026 07:35:47 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/07/2026 07:35:47 AM UTC [INFO] Default storage config from KContext: kci-storage 01/07/2026 07:35:47 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/07/2026 07:35:47 AM UTC [INFO] KContext get_storage_config returned: True 01/07/2026 07:35:47 AM UTC [INFO] KContext storage initialization successful: True 01/07/2026 07:35:47 AM UTC [INFO] Watchdog started with 600s timeout 01/07/2026 07:35:47 AM UTC [DEBUG] Node channel sub id: 102446 01/07/2026 07:35:47 AM UTC [DEBUG] Retry channel sub id: 102447 01/07/2026 07:35:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:35:47 AM UTC [INFO] Press Ctrl-C to stop. 01/07/2026 07:35:47 AM UTC [INFO] Listening for available checkout events 01/07/2026 07:35:47 AM UTC [INFO] Press Ctrl-C to stop. 127.0.0.1 - - [07/Jan/2026 07:36:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:36:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:37:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:37:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:38:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:38:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:39:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:39:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:40:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:40:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:41:15] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 745, in _run_scheduler if self._should_skip_due_to_queue_depth(runtime, job, platform): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/./src/scheduler.py", line 388, in _should_skip_due_to_queue_depth max_queue_depth = runtime.config.max_queue_depth ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'RuntimeLAVA' object has no attribute 'max_queue_depth' 127.0.0.1 - - [07/Jan/2026 07:41:45] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:42:15] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:42:46] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [07/Jan/2026 07:43:16] "GET /health HTTP/1.1" 200 - 01/07/2026 07:43:40 AM UTC [INFO] Stopping. 01/07/2026 07:43:40 AM UTC [INFO] Unsubscribing: 102446 01/07/2026 07:43:40 AM UTC [INFO] Unsubscribing: 102447