Looking at the documentaion for os2trace I found this.
The first feat, trace enabling, is accomplished by
processing each entry in the table of strings within the
executable file that contains the names of the imported
DLLs. If the entry represents one of the supported OS/2
DLLs and the user requested enablement of API tracing for
this DLL, the entry is replaced with the name of the
appropriate trace DLL.
NOTE: Trace enablement alters the contents of the table of
strings within the executable file that contains the names
of the imported DLLs. Although this action does not affect
the functionality of the executable, it does affect its
date and time stamp unless the -P option is specified.
This behaviour is fine if
I'm debugging something, it's not something I'd expect an application, particularly a text editor, to be doing behind my back. It'll also give an antivirus program conniptions. If all you want is to check if an arbitrary application uses only vio calls for output then what you are trying to detect is
a) Is there any output on stdio (some programs use overlaying of vio and standard i/o in which case you don't care about vio)
b) If there is no output on stdio has the program written to the screen using vio calls
in case b if the program has written to the screen using vio calls the cursor will have moved.
So, something along these lines would give you the behaviour you want.
1 Get current cursor position (VioGetCurPos) and save it
2 Write command to pipe
3 Check for output on stdout, if there is you're done
4 If there isn't check the cursor position and compare it to the original, if it's different pop up the vio window, otherwise there was no output.
You can, and indeed you should, check for the case where there's no output on stdout but there is on stderr and take apropriate action(popping up the vio window at a minimum). While it's possible that an application might use vio calls exclusively for error output rather than writing to stderr I've never seen it, nor can I think of a case where it would be sensible.