Thank you for your answers. DosQueryAppType should fit my purpose. This API call refers to the bits Dave mentioned. I assume that the following bits are interesting for me:
001 - (FAPPTYP_NOTWINDOWCOMPAT, 0x00000001): Application type is not-window-compatible.
010 - (FAPPTYP_WINDOWCOMPAT, 0x00000002): Application type is window-compatible.
011 - (FAPPTYP_WINDOWAPI, 0x00000003): Application type is window-API
So it seems that 0x1 means the program runs in the cmd.exe session.
Bit 5 (FAPPTYP_DOS, 0x00000020): Set to 1 if the executable file is in PC/DOS format. Bits 0, 1, 2, 3, and 4 will be set to 0.
DOS programs have to run in their own session, of course. So I think simply checking bit 0x1 set and 0x2 unchecked should be sufficient.
I will report here after testing different sample programs. I am not clear what "window-compatible" means.