So I posted about this particular problem myself over a year ago and I saw somebody else that I think had the same problem. I finally ran across this thread:
https://forums.virtualbox.org/viewtopic.php?f=8&t=90446 and found what seems to be the problem. For some reason on macOS, the scaling inside of VirtualBox to make a non-HiDPI guest usable on a HiDPI ("Retina") screen is very slow. If you tell the OS that VirtualBox isn't HiDPI capable, the OS itself will handle the scaling, and it's MUCH faster.
https://digiajay.com/2022/03/05/macos-monterey-virtualbox-slow-performance-issue/ has a write up on how to make the change. Make sure you change the VM scaling from the default 200% or 250% back to 100% first! This completely fixed the issue for me!
