[LBBS-83] Node deadlock on shutdown

← All Issues || New Issue →

Sort Normal || Priority

Details

Not enough clues here to narrow the cause down, since these functions could be called from a variety of places.

A solution that wouldn't fix the underlying issue but increase visibility would pass the parent calling location into __bbs_mutex_lock and friends, so we have a more useful location.

[2024-11-23 14:37:39.686]   DEBUG[1042872]: socket.c:1678 bbs_node_poll: Node 1: poll returned 1 but got POLLNVAL
[2024-11-23 14:37:49.580] WARNING[1042872]: node.c:370 bbs_node_lock: Spent 19 seconds so far waiting for mutex &node->lock, possible deadlock? (Mutex acquired at node.c:370 10 s ago by LWP 906982)
[2024-11-23 14:37:49.582]   ERROR[1042872]: backtrace.c:267 bbs_log_backtrace: Got 7 backtrace records
[2024-11-23 14:37:49.746]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  0: lbbs bbs_log_backtrace()
[2024-11-23 14:37:49.746]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  1: lbbs __bbs_mutex_lock()
[2024-11-23 14:37:49.746]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  2: lbbs bbs_node_menuexec()
[2024-11-23 14:37:49.747]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  3: lbbs bbs_node_handler()
[2024-11-23 14:37:49.747]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  4: lbbs <unknown>()
[2024-11-23 14:37:49.747]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  5: [0x7efc0f5121c4] libc.so.6                         pthread_create.c:442   start_thread()
[2024-11-23 14:37:49.747]   ERROR[1042872]: backtrace.c:291 bbs_log_backtrace:  6: [0x7efc0f59285c] libc.so.6                                 clone3.S:83    clone3()
[2024-11-23 14:37:51.300]   DEBUG[906991]: mod_smtp_delivery_external.c:1304 run_queue: Processing mail queue (0 messages)
[2024-11-23 14:38:06.246] WARNING[907002]: node.c:196 bbs_node_ip_count: Spent 36 seconds so far waiting to rdlock &nodes, possible deadlock? (rwlock acquired at node.c:812 30 s ago by LWP 906982)
[2024-11-23 14:38:06.247]   ERROR[907002]: backtrace.c:267 bbs_log_backtrace: Got 7 backtrace records
[2024-11-23 14:38:06.399]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  0: lbbs bbs_log_backtrace()
[2024-11-23 14:38:06.399]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  1: lbbs __bbs_rwlock_rdlock()
[2024-11-23 14:38:06.399]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  2: lbbs bbs_node_ip_count()
[2024-11-23 14:38:06.400]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  3: lbbs <unknown>()
[2024-11-23 14:38:06.400]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  4: lbbs <unknown>()
[2024-11-23 14:38:06.400]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  5: [0x7efc0f5121c4] libc.so.6                         pthread_create.c:442   start_thread()
[2024-11-23 14:38:06.400]   ERROR[907002]: backtrace.c:291 bbs_log_backtrace:  6: [0x7efc0f59285c] libc.so.6                                 clone3.S:83    clone3()

Comments

You must be logged in to leave a comment.