I don't believe you'll get there with VMware Workstation as the virtualization platform, as VMware does not provide "guest tools" for OS/2 or ArcaOS. For what can be done within VMware, look at ArcaOS's page:
arcanoae.com/wiki/arcaos/installation-planning/virtual-machine-configuration/
If you switch to VirtualBox (free from Oracle), they do provide a reasonable set of guest tools, and these should work for ArcaOS (they did in my testing). Furthermore, these additional web pages discuss "the blonde guy's" efforts for video improvements under ArcaOS, and additional ArcaOS tweaks for VirtualBox. Perhaps these will get you closer:
blondeguy.com/computer/sodium.html (which then refers to another ArcaOS page)
arcanoae.com/wiki/arcaos/post-install-tips-hints/other-oracle-vm-virtualbox-tweaks/
ArcaOS will provide the guest tools when you install ArcaOS under VirtualBox ... if you're instead running Warp or thereabouts under VirtualBox, you'll need to hunt down the older VirtualBox guest tools for OS/2, provided by NetLabs.org, at:
trac.netlabs.org/vbox
Additionally, there are further tweaks out there to use Virtualbox tools to manage GPU settings in a vm (if Windows is the Host OS) ... something like:
vboxmanage modifyvm "YourVMName" --vram 256
and pointing Virtualbox.exe at a dedicated Nvidia display adapter. I have only lightly tested some of this, and I did manage to get a VM with improved graphics capability, but I didn't test a bunch of cases. Hopefully, the internet will provide those tweak explanations for you. Again, that's with Virtualbox, which does have a level of guest tools support for OS/2 and ArcaOS.
Hope this helps ...