Here's the full instructions on what I did:
1) Shut down all VBox VMs, but leave VBox itself running.
2) On a command line window, run:
VBoxManage.exe setextradata global GUI/MaxGuestResolution any
(on Linux, leave out the .exe of course)
( Or inside of VBox, Preferences->Display, Maximum Guest Screen Size to "None" ).
3) On the same command line window, run:
VBoxManage.exe setextradata "OS/2" "CustomVideoMode1" "1920 x 1200 x 24"
(change OS/2 to match the VM name)
4) Power on the VM and wait for the desktop to finish loading
5) At the command line again, run:
VBoxManage.exe controlvm "OS/2" setvideomodehint 1920 1200 24
( Yes, there's no consistency in the formatting of the resolution between the two commands, it's annoying )
6) Back inside the VM, go into the Screen settings and 1920x1200x16777216 should now show up.
This assumes you have the VBox's version of GENGRADD.DLL loaded following either the instructions with the GAs, or by selecting guest additions in the ArcaOS installer.
Basically, #2 lets you set custom resolutions (they won't work without it), #3 adds the new resolution, and #5 tells the VM about it.