This is normal for OS/2 on modern machines. BIOS is not correctly emulated.
viop.sys might help for DOS/Windows full screen. Not sure where to get it.
Waiting for ArcaOS 5.1 UEFI, where both DOS and Windows will be working full screen and in a window.
Depending on how old ArcaOS version, it might be worth installing the updates. Fixes to drivers seemed to help on some systems.