CE 2.9 – kea and unbound stuck waiting on each other

I'm running pfsense ce 2.9.0 with KEA, Unbound, and PFBlockerNG in Unbound Python Mode. My configuration in 2.8.1 was the same and I did not have any issues. It only started after upgrading to 2.9 and seems to occur roughly once every week or so.

The problem: I have twice now since the upgrade had unbound exceed its queue limits and become completely unresponsive to any/all attempts to query it on the local LAN. This only seems to affect the LAN, not my wireguard tunnels or any other interfaces.

tcp6 0/0/256 ::1.53 tcp4 0/0/256 127.0.0.1.53 tcp6 0/0/256 fe80::2e0:67ff:f.53 tcp6 0/0/256 fe80::2e0:67ff:f.53 tcp4 0/0/256 10.254.0.1.53 tcp4 0/0/256 10.252.0.1.53 tcp4 0/0/256 10.251.0.1.53 tcp4 0/0/256 192.168.10.1.53 tcp6 385/0/256 2600:1700:10b0:d.53 tcp4 385/0/256 192.168.1.1.53 

This is even after implementing kernel tweaks to increase queue limits from this thread: https://www.reddit.com/r/pfBlockerNG/comments/1u399ba/what_is_the_ideal_setup_to_avoid_cpu_spikes_and/

Unlike that thread I don't have a ton of lists in pfblocker or a huge environment. I'm a home user with ~100 clients and only four lists. StevenBlacks Basic ads list, a DOH DNS blocklist, and an ipv4 & ipv6 DOH blocklists.

I've got firewall rules to block external DNS and/or reroute DNS queries back to the firewall as appropriate so clients (except a select few that neeed it) can't bypass the router for lookups. I don't think this is related as again I've had these rules for years and it wasn't an issue until I upgraded to 2.9.

Looking at the output of PS, unbound and kea appear to be waiting on each other in a deadlock:

ps -o pid,state,wchan,%cpu,%mem,command -ax | grep '[u]nbound' 31413 I sbwait 0.0 0.2 /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf status 41507 Is uwait 0.0 4.8 /usr/local/sbin/unbound -c /var/unbound/unbound.conf 47411 I wait 0.0 0.0 /bin/sh /conf/kea4_scripts.d/kea2unbound.sh hook_load 47663 I piperd 0.0 0.5 php /usr/local/bin/kea2unbound --kea-conf /usr/local/etc/kea/kea-dhcp4.conf --unbound-conf /var/unbound/unbound.conf --include-file /var/unbound/leases/l 68064 I sbwait 0.0 0.2 /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf fast_reload 78110 S kqread 0.0 0.2 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf 

I am not an expert in pfsense processes so I had gemini help me diagnose what the different states from the ps command mean exactly:

-PID 41507 (unbound): State is uwait (userspace mutex lock wait). Unbound's main thread is deadlocked waiting on an internal lock. -PID 47411 (kea2unbound.sh hook_load) & PID 47663 (kea2unbound): Kea's dynamic DHCP hook triggered. -PID 68064 (unbound-control ... fast_reload): State is sbwait (socket buffer wait). Kea called unbound-control to push new DHCP lease data via fast_reload, but the command is stuck waiting on Unbound's socket. -PID 31413 (unbound-control ... status): Also stuck in sbwait behind the deadlocked reload. 

requiring me to run the below to restore functionality:

killall -9 unbound-comtrol php unbound pfSsh.php playback svc restart unbound 

Gemini is rather insistent that I should disable early dns registration to work around the issue but I'd rather not unless there is no other way. It's my understanding that disabling that would mean any statically mapped clients would not be registered in DNS until they actually request a new lease from the router. Leaving me with windows of time where attempts to resolve certain hostnames will fail.

Anecdotally I do have a remote router on 2.9 with KEA enabled but no PFBlocker and I do not have this issue. So if the community thinks this is a pfblocker problem rather than a pfsense problem I can move my post over there. But like I said at the start I had these same settings in 2.8.1 and never experienced this so I'm thinking this is a bug rather than a specific issue with my configuration or pfblocker. I haven't seen anything on redmine but I may not be using the proper keywords in my searching.

I also find it rather odd that this isn't a more frequent (like daily) occurence. I guess it requires a perfect storm of conditions for them to get stuck like this?

I'd be interested to hear if anyone else is having this problem and what you did/are doing to resolve it.