OS2 World Community Forum

OS/2, eCS & ArcaOS - Technical => Programming => Topic started by: Martin Vieregg on February 25, 2020, 06:49:43 pm

Title: Observing if a program uses vio calls
Post by: Martin Vieregg on February 25, 2020, 06:49:43 pm
In the german forum os2.org (https://www.os2.org/viewtopic.php?f=6&t=2075) I started a thread with a problem for my PM commandline frontend, which only supports stdin and stdout, but no VIO calls. If a program writes directly into the screen buffer, my PM window does not show it. So the user has to switch between VIO and PM stdout window - I have got a button in my PM window doing that. It would be useful if the program recognizes automatically if a program which is executed via cmd.exe writes VIO output, so the VIO window could get visible automatically.

My first idea was to analyse the EXE binary and search the string VIOCALLS. In the thread, I was told that there is a better way. The program exehdr.exe from the OS/2 2.45 Toolkit in combination with the following rexx script
Code: [Select]
/* analyse executable import viocalls */
parse upper arg efn SuperfluousArguments
if (efn == '') then return

ifn = 'exehdr.txt'

'@e:\os2ddk\base32\tools\os2.386\bin\exehdr /v 'efn' > 'ifn

needle = ' imp '
do while lines(ifn) <> 0
  haystack = linein(ifn)
  there = pos(needle,haystack)
  if (there > 0) then do
    line = substr(haystack,there+length(needle))
    if (left(line,8)=='VIOCALLS') then say line
    end
  end

return

shows not only the fact that a program uses VIO calls, it shows also the number. The list of VIO calls is here:
Code: [Select]
VIOENDPOPUP             VIOCALLS   1
VIO16ENDPOPUP           VIOCALLS   1
VIOGETPHYSBUF           VIOCALLS   2
VIO16GETPHYSBUF         VIOCALLS   2
VIOGETANSI              VIOCALLS   3
VIO16GETANSI            VIOCALLS   3
VIOSETANSI              VIOCALLS   5
VIO16SETANSI            VIOCALLS   5
VIODEREGISTER           VIOCALLS   6
VIO16DEREGISTER         VIOCALLS   6
VIOSCROLLUP             VIOCALLS   7
VIO16SCROLLUP           VIOCALLS   7
VIOPRTSC                VIOCALLS   8
VIO16PRTSC              VIOCALLS   8
VIOGETCURPOS            VIOCALLS   9
VIO16GETCURPOS          VIOCALLS   9
VIOWRTCELLSTR           VIOCALLS  10
VIO16WRTCELLSTR         VIOCALLS  10
VIOPOPUP                VIOCALLS  11
VIO16POPUP              VIOCALLS  11
VIOSCROLLRT             VIOCALLS  12
VIO16SCROLLRT           VIOCALLS  12
VIOWRTCHARSTR           VIOCALLS  13
VIO16WRTCHARSTR         VIOCALLS  13
VIOSETCURPOS            VIOCALLS  15
VIO16SETCURPOS          VIOCALLS  15
VIOSCRUNLOCK            VIOCALLS  18
VIO16SCRUNLOCK          VIOCALLS  18
VIOWRTTTY               VIOCALLS  19
VIO16WRTTTY             VIOCALLS  19
VIOGETMODE              VIOCALLS  21
VIO16GETMODE            VIOCALLS  21
VIOSETMODE              VIOCALLS  22
VIO16SETMODE            VIOCALLS  22
VIOSCRLOCK              VIOCALLS  23
VIO16SCRLOCK            VIOCALLS  23
VIOREADCELLSTR          VIOCALLS  24
VIO16READCELLSTR        VIOCALLS  24
VIOSAVREDRAWWAIT        VIOCALLS  25
VIO16SAVREDRAWWAIT      VIOCALLS  25
VIOWRTNATTR             VIOCALLS  26
VIO16WRTNATTR           VIOCALLS  26
VIOGETCURTYPE           VIOCALLS  27
VIO16GETCURTYPE         VIOCALLS  27
VIOSAVREDRAWUNDO        VIOCALLS  28
VIO16SAVREDRAWUNDO      VIOCALLS  28
VIOGETFONT              VIOCALLS  29
VIO16GETFONT            VIOCALLS  29
VIOREADCHARSTR          VIOCALLS  30
VIO16READCHARSTR        VIOCALLS  30
VIOGETBUF               VIOCALLS  31
VIO16GETBUF             VIOCALLS  31
VIOSETCURTYPE           VIOCALLS  32
VIO16SETCURTYPE         VIOCALLS  32
VIOSETFONT              VIOCALLS  33
VIO16SETFONT            VIOCALLS  33
VIOMODEUNDO             VIOCALLS  35
VIO16MODEUNDO           VIOCALLS  35
VIOMODEWAIT             VIOCALLS  37
VIO16MODEWAIT           VIOCALLS  37
VIOGETCP                VIOCALLS  40
VIO16GETCP              VIOCALLS  40
VIOSETCP                VIOCALLS  42
VIO16SETCP              VIOCALLS  42
VIOSHOWBUF              VIOCALLS  43
VIO16SHOWBUF            VIOCALLS  43
VIOSCROLLLF             VIOCALLS  44
VIO16SCROLLLF           VIOCALLS  44
VIOREGISTER             VIOCALLS  45
VIO16REGISTER           VIOCALLS  45
VIOGETCONFIG            VIOCALLS  46
VIO16GETCONFIG          VIOCALLS  46
VIOSCROLLDN             VIOCALLS  47
VIO16SCROLLDN           VIOCALLS  47
VIOWRTCHARSTRATT        VIOCALLS  48
VIO16WRTCHARSTRATT      VIOCALLS  48
VIOGETSTATE             VIOCALLS  49
VIO16GETSTATE           VIOCALLS  49
VIOPRTSCTOGGLE          VIOCALLS  50
VIO16PRTSCTOGGLE        VIOCALLS  50
VIOSETSTATE             VIOCALLS  51
VIO16SETSTATE           VIOCALLS  51
VIOWRTNCELL             VIOCALLS  52
VIO16WRTNCELL           VIOCALLS  52
VIOWRTNCHAR             VIOCALLS  53
VIO16WRTNCHAR           VIOCALLS  53
VIOASSOCIATE            VIOCALLS  55
VIO16ASSOCIATE          VIOCALLS  55
VIOCREATEPS             VIOCALLS  56
VIO16CREATEPS           VIOCALLS  56
VIODELETESETID          VIOCALLS  57
VIO16DELETESETID        VIOCALLS  57
VIOGETDEVICECELLSIZE    VIOCALLS  58
VIO16GETDEVICECELLSIZE  VIOCALLS  58
VIOGETORG               VIOCALLS  59
VIO16GETORG             VIOCALLS  59
VIOCREATELOGFONT        VIOCALLS  60
VIO16CREATELOGFONT      VIOCALLS  60
VIODESTROYPS            VIOCALLS  61
VIO16DESTROYPS          VIOCALLS  61
VIOQUERYSETIDS          VIOCALLS  62
VIO16QUERYSETIDS        VIOCALLS  62
VIOSETORG               VIOCALLS  63
VIO16SETORG             VIOCALLS  63
VIOQUERYFONTS           VIOCALLS  64
VIO16QUERYFONTS         VIOCALLS  64
VIOSETDEVICECELLSIZE    VIOCALLS  65
VIO16SETDEVICECELLSIZE  VIOCALLS  65
VIOSHOWPS               VIOCALLS  66
VIO16SHOWPS             VIOCALLS  66
VIOGETPSADDRESS         VIOCALLS  67
VIO16GETPSADDRESS       VIOCALLS  67
VIOGLOBALREG            VIOCALLS  70
VIO16GLOBALREG          VIOCALLS  70
XVIOSETCASTATE          VIOCALLS  71
XVIO16SETCASTATE        VIOCALLS  71
XVIOCHECKCHARTYPE       VIOCALLS  72
XVIO16CHECKCHARTYPE     VIOCALLS  72
XVIODESTROYCA           VIOCALLS  73
XVIO16DESTROYCA         VIOCALLS  73
XVIOCREATECA            VIOCALLS  74
XVIO16CREATECA          VIOCALLS  74
VIOCHECKCHARTYPE        VIOCALLS  75
VIO16CHECKCHARTYPE      VIOCALLS  75
XVIOGETCASTATE          VIOCALLS  76
XVIO16GETCASTATE        VIOCALLS  76

But this solution anlyses only the program code and I do not know if the API calls are really exeuted. For example, XCOPY uses VIO calls, but only in very special circumstances I have not found out yet.

There is a program os2trace (http://www.angelfire.com/tx4/blaschke/OS2Trace/)  which excatly makes the job I need: Observing an application and journaling which API calls it calls.

The perfect solution would be if my PM program observes the EXE file which is executed via cmd.exe in the known session which is controlled by my program like os2trace does. Is the author Dave Blaschke still acitive? Perhaps he or someone can explain or help me with the code I need. I would need a simple C or Pascal commandline program which observes a specific binary and gets an output "VIOCALL used!" if a VIOcall is sent. Then I could copy these lines into my PM program in an own thread.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on February 26, 2020, 01:53:12 pm
Looking at the documentaion for os2trace I found this.
Quote
The  first  feat,   trace  enabling,   is  accomplished  by
                 processing  each  entry in  the table of strings within the
                 executable  file that  contains the  names of  the imported
                 DLLs.  If the entry  represents one  of the  supported OS/2
                 DLLs and the  user requested  enablement of API tracing for
                 this  DLL,  the  entry is  replaced  with the  name  of the
                 appropriate  trace  DLL.   

Quote
   NOTE:  Trace enablement alters the contents of the table of
                 strings within the executable file  that contains the names
                 of the imported DLLs.  Although this action does not affect
                 the  functionality of  the executable,  it does  affect its
                 date and time stamp unless the -P option is specified.

This behaviour is fine if I'm debugging something, it's not something I'd expect an application, particularly a text editor, to be doing behind my back. It'll also give an antivirus program conniptions. If all you want is to check if an arbitrary application uses only vio calls for output then what you are trying to detect is
a) Is there any output on stdio (some programs use overlaying of vio and standard i/o in which case you don't care about vio)
b) If there is no output on stdio has the program written to the screen using vio calls
in case b if the program has written to the screen using vio calls the cursor will have moved.

So, something along these lines would give you the behaviour you want.
1 Get current cursor position (VioGetCurPos) and save it
2 Write command to pipe
3 Check for output on stdout, if there is you're done
4 If there isn't check the cursor position and compare it to the original, if it's different pop up the vio window, otherwise there was no output.

You can, and indeed you should, check for the case where there's no output on stdout but there is on stderr and take apropriate action(popping up the vio window at a minimum). While it's possible that an application might use vio calls exclusively for error output rather than writing to stderr I've never seen it, nor can I think of a case where it would be sensible.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on February 26, 2020, 05:36:53 pm
Laurence,

yes, that idea is very useful for my purpose ! VioGetCurPos requires a VIO handle. I have got a window handle and a session ID. How can I get the VIO handle?
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on February 26, 2020, 06:21:42 pm
Quote
VioHandle (HVIO) - input
Presentation-space handle.

This must be zero, unless the caller is a Presentation Manager application; in this case, it must be the value returned by VioCreatePS
Your helper application isn't a presentation manager application, so the VioHandle should be zero.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on February 26, 2020, 10:56:27 pm
Quote
Your helper application isn't a presentation manager application, so the VioHandle should be zero.

Ah, yes, the helper application! Then I can simply use semaphores again. So I see three ways for doing the job:

(1) I create a thread in the helper program which always queries VioGetCurPos (e.g. 4 times every second, with a DosSleep in the loop) and observes the changes of the cursor position. If a change has been observed, it sends a message via semaphores to the PM application.

(2) The timer (not an additional thread) is in the PM application and it sends a query to the helper via semaphores. This query is only sent if some conditions are true
(a) a command is running (not prompt state)
(b) the application uses VIO calls
(c) no output is received for e.g. 1/4 or 1/2 seconds.
I think it won't be necessary to check all conditions. The helper remembers the last query and compares it with the new query. The helper sends the result back to the PM application via semaphores.

(3) In the case of the possibility that the PM helper can query directly the VIO handle (from "outside"), I would prefer the short way without helper app because it is easier to program.

But anyway, it seems that a clean solution will be definitely possible! What solution would be preferable in your opinion? Option (2) has different possibilities. So we have to discuss (b) if a VIO call analyzation of the binary is still necessary or can be resigned, because the curorpos query via helper needs less processor power.  Also condition (c) is to discuss. If a program generates both VIO output and stdout, it would be better to show both windows? It can happen that an application creates very fast some lines of VIO output and ends again. Perhaps some user settings would be helpful.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on February 27, 2020, 01:13:16 am
For 2b and 2c have the thread that writes the command to stdin set a semaphore and have the thread that reads from stdout post that semaphore, that way it's only active when a command is running, so you don't have to handle 2a.  Then have a thread that is started immediately after the command is written to stdin which waits on the semaphore being posted with a timeout value. If your reader thread reads data from stdin then it should post the semaphore, The thread waiting on the semaphore should call DosQueryEventSemaphore to get the post count to distinguish between the semaphore being posted or the timeout expiring. If the timer has expired it should check whether the cursor position has changed if it has then pop up the vio window, if it hasn't there's been no output.

I'm assuming that the helper application sits in a DosWaitEventSemaphore which the PM application posts when it sends a command.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on February 27, 2020, 11:08:13 am
OK. I see it clear now and I think this should work:

1. Storing VIO cursor position before exeuting the command
The user presses ENTER in the editor main thread to send the new command to cmd.exe. The editor posts a semaphore "helper, please tell me the current VIO cursor position". I can use the semaphore I already use for the Ctrl-C exception because both posts are from the editor main thread. Then the helper queries the cursor position via "VioGetCurPos" and sends the information immediately back to the editor thread which waits for a second new semaphore in a thread. The data is hold in Shared Memory which I already have. Now the editor sends the user input to cmd.exe stdin (and not before receiving the cursor position from the helper to avoid a race problem).

2. Observing while executing
In a specific frequency (e.g. I have got already a timer with 250 ms), the editor asks the helper about the current cursor position in the VIO window (same procedure as 1.). If the x or y values have changed, the VIO window gets popped up.
This query only gets executed if there is currently no output received in the editor and/or if an EXE file is executed which has got VIO calls.

3. Observing when the command execution ends
If a prompt is shown again in the editor and the current command has ended execution, a last query of the VIO cursor position gets started. If there's a change of the VIO cursor position, the VIO window gets popped up. While the editor waits for user input (prompt state), no timer query to the helper proceeds.

So for me it seems, that in Point 2., it is sufficient to observe a coming input, so analyzing the EXE binaries gets obsolete which is a lot of problematic code (observing the execution of cmd files) I do not need anymore and is not programmed yet anyway.

In my opinion, this is a real perfect solution. If the executed command creates both VIO and Stdout, the VIO window is shown in any case when ending the command execution. In this case, I can program the VIO window position beneath the editor window so the user can see both outputs. So it is absolutely precluded that the user does not see a VIO output.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on February 27, 2020, 12:34:06 pm
1. The PM app shouldn't care about the cursor position, the PM app should care about output. Use  semaphores to indicate that the command has completed and produced output, The helper app posts a semaphore when the command has completed, the helper app should check for vio output if the command doesn't produce output on stdout and indicate this to the PM app, which should pop up the vio window if it's appropriate. It could indicate the presence of vio output using a semaphore or by writing a specific value to the pipe before writing the prompt to the pipe.

2. Don't observe while executing, the PM app should care about output, not how  the shell is doing what it is doing, that's the province of the helper application. Either it's got output to display or it hasn't and the output is either in the pipe or it needs to pop up a vio window. Which action is required should be indicated by the helper application, not computed by the PM application.   Modularity breeds simplicity.

3. See 1.

Analysing the executable files is unnecessary, and certainly not something that a shell front end should be doing. It's an interesting project in it's own right though. If you are interested in pursuing it then the OS/2 debugging handbook is a good place to start.
https://openlibrary.org/works/OL8414318W/The_OS_2_Debugging_Handbook (https://openlibrary.org/works/OL8414318W/The_OS_2_Debugging_Handbook)
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on February 27, 2020, 12:58:51 pm
Quote
2. Don't observe while executing, the PM app should care about output

Laurence, so in your opinion the loop/timer which continously queries the VIO cursor position should be placed in the helper app. Because non-PM programs do not know timer events, I have to create a thread with a DosSleep loop and continous querying VioGetCurPos. This only proceeds if the PM editor tells the helper doing so. If the prompt appears again in the PM editor, the editor tells the helper "tell me if the VIO cursor position has changed and end the loop."
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on February 27, 2020, 09:26:33 pm
You shouldn't be continously polling the Vio cursor position, it's uneccessary. The only time you need to know if the Vio cursor has moved is following the completion of a command that produces no output on stdout. The helper program should do the checking and indicate to the PM application that there is Vio output so the PM application can pop up the Vio window.

The helper application should
1) check the position of the Vio vursor
2) write the command to stdin
3) Check for output from the command on stdout
4) Indicate to the PM app that the command has completed
5) If there is output on stdio send it to the PM app for display
6) If there is no output on stdio check if the Vio cursor position has changed
7) If the Vio cursor position has changed indicate to the PM app that there is Vio output.

The PM app should
1) Send the command to the helper app.
2) Await indication that the command has completed
3) Check if output has been sent from the helper program. If there is display it. You're done
4) If there's no output check if the helper has indicated that there's Vio output. If there is pop up the Vio window.

For 4) in the PM app this could be done by either the helper app writing a sentinel value to the pipe (something like "**vio_output**" would probably work) or posting an event semaphore. If you use a sentinel value then it's simply a check for that value at the start of the display routine which forks to popping up the vio window instead of displaying the text read from the pipe. If you use an event semaphore then you need a thread waiting on the event semaphore that  pops up the vio window when the helper posts the event semaphore.

The PM app shouldn't be doing anything with the Vio cursor, it shouldn't be going near it, and polling it  using a timer is just broken. If the command takes a long time to complete you'll run the risk of polling too early, at which point your PM app will read that there's no change in the Vio cursor and fail to pop up the window. It's a race condition, and that's why you should use semaphores and not polling.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on February 28, 2020, 10:09:00 am
Laurence, thank you for reasoning. Most things are congruent with my ideas, but I think that you have one circumstance not in view.
Quote
You shouldn't be continously polling the Vio cursor position, it's uneccessary. The only time you need to know if the Vio cursor has moved is following the completion of a command that produces no output on stdout.
If a VIO program waits for user input (e.g. a VIO-based editor or a menu program like DFSEE), I need a loop with continous querying. After again thinking about the problem, I have got this result: The PM editor has to send two kinds of queries to the helper:
1. A single query after a current command execution ends and a prompt is shown again: "Please tell me if VIO (cursor position) has changed and end the loop, if it is running".
2. A continous query via thread loop: "Please observe the VIO window and tell me immediately if VIO has changed."

The continous query is only sent to the helper if no stdout is shown for a specific time, e.g. 300 or 400 ms.
An option to optimize is to check whether the output contains Returns without Carriage (^M only without ^J). This output is typical for progress bars of stdout programs. In this case, a continous VIO observation is not necessary in my opinion. The editor knows it already.

This is all quite simple to program for me, because a lot of stuff can be reused: For the post from the PM window to the helper, I can use the semaphore and Shared Memory I use for the Ctrl-C exception. Then in the PM editor I have one unused thread left which starts the helper. I can add here a wait for semaphore loop. I have not to submit any data from the helper to the PM editor, I have only to wait for the semaphore event "pop VIO window now". This semaphore event is used in both cases 1. and 2.
So I will need a new semaphore. I will generate the semaphore fin the PM editor, the pointer value is submitted via commandline parameter to the helper again. Then I need a new thread in the helper executing the loop.

Two minor questions:
In the helper, I can recreate the thread if needed or suspend/resume the old thread. (Any ideas what is to prefer?)
When creating the new second semaphore, I hope that it is allowed that the receiver creates the semaphore? Normally the sender creates it. The way via commandline parameters to submit the pointer value is very easy for me.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on February 28, 2020, 03:06:27 pm
Quote
2. A continous query via thread loop: "Please observe the VIO window and tell me immediately if VIO has changed."

The continous query is only sent to the helper if no stdout is shown for a specific time, e.g. 300 or 400 ms.
Can you tell  me what the use case for this is? When would the Vio cursor change other than when there is output from a command. In which case what you're doing in 1, covers this case.

Recreating the thread is probably simpler, suspending and resuming it means you have to implement some way of checking that the values of any variables haven't changed behind the threads back while it was suspended. If it turns out that recreating the thread becomes a performance bottleneck then you can revisit the question. Given that you're processing commands that are being typed by the user it's unlikely that anyone can type fast enough to make that an issue.

The receiver creating the semaphore is the most sensible way to do this in this context. Think of it in terms of the first process, the PM application,  setting up the environment in which the helper process will execute rather than which process posts and which process waits on the semaphore.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on February 28, 2020, 04:30:08 pm
If I exeute a binary directly, in most cases there comes VIO output immediately (I have no sample, but it can be different, too, that it first generates some stdout output "starting my app... please wait" and then switches to VIO, and after finishing VIO, some stdout is created (which wouldn't be a problem for me). In most cases, a first query after 400 ms without stdout output should be sufficient.

More sophisticated is the case of executing a cmd file. We discussed the case of sending the cmd file content line by line to cmd.exe stdin and the PM editor checks whether there's a VIO program in the sent text line or not. (We want to avioid this procedure.) I have got big cmd files in use which runs two minutes. It can happen that in such a long cmd file, a VIO program is placed between other stdout programs and commands.

Meanwhile, I have doubt, too, that a loop within the helper app is really useful. Instead, the PM editor could send a single query "VIO cursor position changed?" after 300-400 ms after the last received stdout. And if nothing happens at all - in the case the helper detects no VIO program and there's no stdout output anyway for seconds - the editor does not query anymore. Only if there's a new stdout output and a pause again, then it queries again. So the case of a cmd file should be covered.

In this case, I do not need a thread in the helper program which simplifies the code. No thread, no problem in suspending, resuming, memory access conflicts to consider and so on.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 01, 2020, 09:19:20 am
I began to program now and I have found a suspect difference between headers of VioGetCurPos:

in
http://www.edm2.com/index.php/VioGetCurPos
the second parameter Row is PULONG, but in my header file (Borland C 2.0), the header file is defined as PUSHORT16. What is correct now? Can someone take a look at a current development environment (Open Watcom?) in bsesub.h ? Thank you.
Title: Re: Observing if a program uses vio calls
Post by: Wim Brul on March 01, 2020, 10:30:46 am
I began to program now and I have found a suspect difference between headers of VioGetCurPos:

in
http://www.edm2.com/index.php/VioGetCurPos
the second parameter Row is PULONG, but in my header file (Borland C 2.0), the header file is defined as PUSHORT16. What is correct now? Can someone take a look at a current development environment (Open Watcom?) in bsesub.h ? Thank you.

PULONG for Row in the wiki is definitely  wrong. It must be USHORT. I will change it in the wiki accordingly.

EDIT: Now I am not so sure anymore. I see VioGetCurPos and VioGetCurPos (FAPI) in the wiki. I am used to code my VIO calls as to the FAPI definitions, which is 16 bit source code. That VioGetCurPos (FAPI) definition is correct. It seems to me that the VioCurPos definition applies to 32 bit source code. Hence PULONG seems to be correct for its Row and Column parameters.

EDIT: I know better now. Therefore I have undone my wiki changes.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 01, 2020, 10:38:31 am
From the IBM Toolkit 4.5 Control Program Guide and Reference
Quote
#define INCL_VIO
#include <os2.h>

PULONG     Row;        /*  Row return data. */
PUSHORT    Column;     /*  Column return data. */
HVIO       VioHandle;  /*  Presentation-space handle. */
APIRET     rc;         /*  Return code. */

rc = VioGetCurPos(Row, Column, VioHandle);
Title: Re: Observing if a program uses vio calls
Post by: Wim Brul on March 01, 2020, 11:35:37 am
From the IBM Toolkit 4.5 Control Program Guide and Reference
Quote
#define INCL_VIO
#include <os2.h>

PULONG     Row;        /*  Row return data. */
PUSHORT    Column;     /*  Column return data. */
HVIO       VioHandle;  /*  Presentation-space handle. */
APIRET     rc;         /*  Return code. */

rc = VioGetCurPos(Row, Column, VioHandle);

Yes, that info is there, but when you look into BSESUB.H you will see that only the 16 bit version has been defined in there.     
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 01, 2020, 12:24:54 pm
In the Borland C header, both parameters row and column are PUSHORT. So we have three different meanings. Anyway, the original 2 times PUSHORT definition works. I assume the are problems if values higher than 65536.

Meanwhile, the new mechanism for popping up VIO windows works! But I have found a program TED.EXE (very old minimalistic editor for editing config.sys on 3,5 inches discs) which is not recognized: It is a pure VIO program and when starting, it draws a menu on the bottom of the VIO window and then it awaits input on the topleft cursorposition (0,0). So my program thinks that there was no input. Are there other useful VioGet... calls where I can additionally query and compare other values than the cursor position? Perhaps comparing the whole screen buffer? (Which is not as big today as it was 20 years before...) I could write user settings for these additional queries.
Title: Re: Observing if a program uses vio calls
Post by: Wim Brul on March 01, 2020, 12:50:10 pm
In the Borland C header, both parameters row and column are PUSHORT. So we have three different meanings. Anyway, the original 2 times PUSHORT definition works. I assume the are problems if values higher than 65536.

Meanwhile, the new mechanism for popping up VIO windows works! But I have found a program TED.EXE (very old minimalistic editor for editing config.sys on 3,5 inches discs) which is not recognized: It is a pure VIO program and when starting, it draws a menu on the bottom of the VIO window and then it awaits input on the topleft cursorposition (0,0). So my program thinks that there was no input. Are there other useful VioGet... calls where I can additionally query and compare other values than the cursor position? Perhaps comparing the whole screen buffer? (Which is not as big today as it was 20 years before...) I could write user settings for these additional queries.

Use VioReadCharStr to compare the whole screen buffer (I think 8192 chars max)  or use VioReadCellStr to compare the whole screen buffer including colors.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 02, 2020, 06:34:20 pm
I think I will compare the whole screen buffer (max 2x8192 bytes). If I understand it right, a "cell" is a byte for the ASCII value and a byte for the colors and attributes. VioGetBuf let me show both the length of the buffer and the address. The user can choose whether to compare the Cursor position or the whole screen buffer.

Does anyone know if the screen buffer changes with the "mode" command? Can VIO programs change the window size on their own?
Title: Re: Observing if a program uses vio calls
Post by: Dave Yeo on March 02, 2020, 07:00:50 pm
Does anyone know if the screen buffer changes with the "mode" command? Can VIO programs change the window size on their own?

I was compiling something inwhich the GCC output used ANSI colours and the VIO window unexpectedly changed to about 20 chars wide, I assume due to a ANSI escape sequence.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 02, 2020, 08:11:58 pm
You can check the current mode using VioGetMode, Vio programs can change the mode using VioSetMode.

The length of the buffer returned by VioGetBuf is calculated as the product of the number of columns, rows and cell size. So, if the mode command changes the number of columns or rows the length of the buffer will also change.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 03, 2020, 08:57:41 am
First, VioGetBuf has in the edm2 wiki the same wrong description. Length is USHORT and not ULONG. It seems that the 8192 char limitation depends on the USHORT values which are spread all over the VIO calls. If I have true color foreground, background, attributes and the ASCII value, then I need 8 bytes for one cell. I think this is the reason for the 8192 chars limit. That means, even if we write the VIO dll new, all existing programs have got the USHORT values in their code. (Perhaps in Windows, this restriction has not existed since Windows NT?) For a perfect solution, we would need a new VIO32 API interface and a compatibility mode. This is a big thing, so I think that the way I have chosen to pop the original VIO window is the right one for the moment.

------------------

But VioGetBuf does not fit my purpose. I want to copy the screen buffer to compare with an ancient state. VioGetMode fits my purpose. I get a structure VIOMODEINFO which holds lots of interesting information. I have logged some values:

VioGetMode returns rc = 0; buffer length 4000 full length 4000 columns 80 rows 25 address 753664

The length of the buffer is 80 x 25 x 2 = 4000. One byte is for the ASCII value, one for the attribute and colors.

But the address of the screen buffer is too low for a normal memory pointer. memcpy let the program crash, the access to this address is not valid.

The definition in VIOMODEINFO is:

ULONG   buf_addr
ULONG buf_length
ULONG full_length


(Interesting: buf length has got here a 32 bit value and this is really true.)

Any idea how to get access to the data? Perhaps I have to convert this value to a real pointer value.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 03, 2020, 09:55:28 am
Quote
I want to copy the screen buffer to compare with an ancient state.
If all you want to do is see if new output has been writtten to the screen then VioReadCharStr will do what you want.
1. Calculate the length of the before buffer you require from the number of rows and columns returned from VioGetMode.
2. Allocate memory for the before buffer
3. Call VioReadCharStr to read the entire screen

Do stuff with command shell

4.  Calculate the length of the after buffer you require from the number of rows and columns returned from VioGetMode.
5. Allocate memory for the after buffer
6. Call VioReadCharStr to read the entire screen

7. Compare the before buffer with the after buffer

8. Clean up.

It's ugly, but it should work..
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 03, 2020, 03:20:46 pm
OK, Laurence. VioReadCharStr works now. TED.EXE which starts at cursorposition (0,0) and ends after writing the screen on cursorposition (0,0) is recognized now. In most cases, only 2000 bytes have to be compared. But anyway, I wonder why I cannot access directly to the buffer address buf_addr of VioGetMode.

Again, the VioReadCharStr length parameter is USHORT, not ULONG. I think the whole VIO EDM2 documentation has to be checked.

Thank you Laurence and Wim again for your valuable hints and explanations !
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 03, 2020, 07:05:06 pm
 
Quote
But anyway, I wonder why I cannot access directly to the buffer address buf_addr of VioGetMode.
There appears to be two distinct versions of the VioGetBuf  call, and some others. The version with the length parameter as a USHORT appears to be the FAPI version, the version with the length parameter as a ULONG appears to be the 32 bit API, which was never distributed as part of the toolkit. It was available as the 32TEXT beta on the developers connection, however it has some fairly serious bugs. The 32 bit Unicode console API from JdeBP is built upon the 32 bit Vio, Kbd and Mou API, but is bundled with dll's which have had the bugs fixed. It can be found on Hobbes as conapi.zip.

The FAPI has some fairly significant limitations.
Quote
Real Mode
    max. 640K memory
    no virtual address space
    no multitasking
    no undocumented OS services
    If the filename of an executable produced by BIND is changed, then it will not run under DOS 2.1.

Protected Mode
    16 MB memory
    1GB virtual address space
http://www.edm2.com/index.php/Family_API (http://www.edm2.com/index.php/Family_API)
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 05, 2020, 09:10:02 am
If I understand the linked EDM2 FAPI page right, VIO is the only part of OS/2 which is still used in 16 bit and 32 bit VIO has started (I assume about the year 2000, at the end of IBM OS/2 development), while all other FAPI stuff has been rewritten in 32 bit and the conversion project has not finished. In the EDM2 documentation, some but not all VIO calls are documentated in two variants, one 16 bit with (FAPI) and one 32 bit without.

http://www.edm2.com/index.php/VioReadCharStr_%28FAPI%29
http://www.edm2.com/index.php/VioReadCharStr

Some parameters are USHORT in the FAPI variant and ULONG in the non-FAPI (32 bit) variant.

So that means the last correction in EDM2 was false. All VIO calls in the EDM2 documentation which have a FAPI correspondent page have to get a link with a sentence: "VIO 32 (bla bla) needs DLL (bla bla) not supported by default, Link to FAPI call"

Some calls have got a correspondent FAPI entry, others not.

Where are the binaries (DLLs) of vio16 and vio32 and do we have the source code to complete the vio32 calls in a long run? (I don't need this stuff for my editor, meanwhile all works fine.)
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 07, 2020, 09:31:28 am
I thaught that I have all VIO specific stuff solved (works really nice and handy, with two different positions of the VIO window dependent on the prompt state of the editor window, so nothing important is covered), but there's another special situation which is not programmed yet and where my program still fails:

[C:\] dir | more

more let rest the output at stdout and writes into stdout (not VIO!) an additional line

----more----

and then it waits for a keypress via VIO. Stdin Input always requires pressing ENTER at the end of the line, and VIO accepts a single keystroke.

Now the behaviour of my program is like I have it programmed:

- The "dir" output is shown in the editor and ends with a "----more----" line
- the VIO window does not pop up because there's no output to show (it is still empty or unchanged)
- but the invisible and empty VIO window now expects a keystroke!

In the editor I distinguish between "prompt state" and "not prompt state" (the user types a new command or the last command is still executed). So I can simply send the user input in not prompt state (program is running) to the VIO window by a WM_CHAR message to the Vio Window handle, I think this should work (have not tried yet). But now I get a new problem: I need to know whether a program waits for stdin input (a character or a line which is finished by typing ENTER) or the VIO window waits for a single keystroke. I have no clear solution for this problem yet. Any hints are appreciated. Again, the helper program running in the VIO window could perhaps help me in observing something.

I would not have the problem if MORE would send the "----more----" line to VIO instead of stdout. The MORE substitute on Hobbes (https://hobbes.nmsu.edu/download/pub/os2/util/browser/more_20b.zip) (a VIO read only editor where you have to press "q" to quit) runs completely in VIO and makes no problems. Of course users of my program do not need MORE anymore, but it could appear in cmd files. And  there's the question if there are other programs which require VIO console input without writing to VIO or MORE is the only case - all stdout programs with a "press ANY key" (not only ENTER) are candidates for this problem.

Of course, I could scan the last stdout line content by searching "more" or "any key" (oops, how much languages do exist?), but I hope we find a general solution like we did for VIO output by observing the whole screen text buffer.

Perhaps one of these calls can help:
http://www.edm2.com/index.php/Category:Kbd
http://www.edm2.com/index.php/Category:Vio

Title: Re: Observing if a program uses vio calls
Post by: Andreas Schnellbacher on March 07, 2020, 11:25:42 am
BTW: EPM produces this output:

Code: [Select]
epm: D:\ > dir |more
SYS0436: An invalid VIO handle was found.
SYS1805: The process tried to write to a nonexistent pipe.

epm: D:\ >
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 07, 2020, 11:35:49 am
EPM does not support VIO. Even the stdin/out support is very poor. When starting a VIO program in EPM, an error message occurs (TED.EXE) or nothing happens (DFSOS2.EXE).
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 07, 2020, 11:55:44 am
Quote
- but the invisible and empty VIO window now expects a keystroke!
No, the process running in that session is expecting a keystroke, it's the MORE command that's expecting the keystroke on stdin, not the vio window. 
Quote
In the editor I distinguish between "prompt state" and "not prompt state"
What you've got is a "not a cmd prompt state", You've got the prompt for the interactive program that's running in the command shell, in the example that's the MORE command. That prompt is being displayed in the editor, so the user knows that input is expected. The named pipe will be empty because the PM program has read the output, including the prompt, from the process running in the cmd shell.

Calling DosPeekNPipe will show that there is no data available in the pipe because the process is waiting for input on stdin. Use that check to indicate that the user's input should be piped to the process awaiting input. I'm assuming that you are checking the return from DosRead for an ERROR_MORE_DATA return code, in order to ensure that there is no data left in the pipe from a partial read.

The user will still need to press return in the editor to actually send the input, which may result in the process receiving an additional keystroke. If the process is expecting single character i/o the process should be flushing the keyboard buffer between prompts, if it isn't that's not your bug, it's the program that you're running in the shell that's buggy.

Title: Re: Observing if a program uses vio calls
Post by: Andi B. on March 07, 2020, 07:49:57 pm
LarsonCommander does fire up a new cmd window in case | more is used with dir.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 07, 2020, 10:47:33 pm
Laurence, it 's all right what you have written, but it does not target my problem. Indeed, the user (not my program) sees that keyboard input is expected. But if the user presses a key on the keyboard, my program does not know how to handle it: sending it to the VIO window via WM_CHAR messages (hope this works) or sending it to cmd.exe via STDIN. In the STDIN variant, a whole line is collected in the editor and is sent in a whole line after the user has pressed ENTER. This problem is independent from the MORE and the empty VIO window problem. I have solved recognizing VIO output, but not recognizing VIO input yet. In most cases, the problem does not occur at the surface, because before expecting VIO input, a VIO output has been recognized and the VIO window has popped on top so it catches the keyboard input anyway. It is only the very specific problem if no VIO output has been generated from a VIO program and a user input via VIO is expected. Perhaps we can simply classify MORE as a bug.

With the speical MORE case, it would be perhaps the best option that the editor itself interpretates the "| more" input and stops with output after one page has been written. Or we write a MORE substitute which is identical to the original one, but does not wait for a VIO keypress and instead, it waits for a stdin line (so the user has to press ENTER instead of any key:

---more---- press Enter to continue

The case of
Code: [Select]
more <filename.txthas the same problem/behaviour.

Can we get the original MORE.EXE source code? (I assume it is quite short.) The usage of MORE is quite variable, so an own executable is perhaps more useful than implementing in the editor (which again forces problems when using in cmd files).
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 08, 2020, 12:38:10 am
The problem with sending a WM_CHAR, or indeed any sort of message is that the cmd process doesn't have a message queue. The frame window does, but that's handling messages for the frame window, the system menu etc.

I have to say that, given the purpose is to get the output of a command into the text editor, I can't see a use case that requires it to be piped through a paginator in the cmd shell. There's no relationship between the dimensions of the cmd shell and the text editor, so it'll end up looking odd,  and, presumaby, the user wants to do some editing of the output text in which case piping the output through a paginator is pointless.

There are a number of implementations of the MORE command, it's one of the GNU utilities, along with LESS, so getting access to source code isn't a problem. LESS is part of the coreutils available via ANPM, and is a far better pager.

An alternative approach would be to deal with piping the output from one command into another within the editor. Instead of passing a line of commands containing pipes to the shell parse the command line and pass each command seperately, collecting the output of each command and using that as the input for the next command in the pipeline. That lets you do your own paginationin the editor, where it makes more sense.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 08, 2020, 03:47:53 pm
Quote
The problem with sending a WM_CHAR, or indeed any sort of message is that the cmd process doesn't have a message queue. The frame window does, but that's handling messages for the frame window, the system menu etc.

Can I perhaps send a message to the helper (the way I have already programmed via Semaphores and Shared Memory) that a user has pressed a key on the keyboard and the helper program writes the character to the keyboard buffer of its own session?

Thinking about my problem has resolved that if no VIO output happened and the user presses SPACE (SPACE is no useful answer for a menu or Y/N query), it is most likely, that the SPACE has to be sent to the VIO window and not to STDIN. This should cover the majority of cases.
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 08, 2020, 03:51:02 pm
Quote
Can I perhaps send a message to the helper (the way I have already programmed via Semaphores and Shared Memory) that a user has pressed a key on the keyboard and the helper program writes the character to the keyboard buffer of its own session?
What you did in that instance was use the semaphore to trigger sending a signal to the helper precisely because the helper didn't have access to the keyboard to get the break signal triggered by pressing ctrl-c on the keyboard. Writing to the screen isn't a keypress, the process running in your shell is waiting for a keypress, Vio programs use KbdCharIn, KbdStringIn or KbdPeek to read key presses and they only have access to the keyboard when they're in focus. For a non PM session you can use a character device monitor to insert data into the character stream from the keyboard.
http://www.edm2.com/index.php/PDDREF:Character_Device_Monitors (http://www.edm2.com/index.php/PDDREF:Character_Device_Monitors)
Quote
Thinking about my problem has resolved that if no VIO output happened and the user presses SPACE (SPACE is no useful answer for a menu or Y/N query), it is most likely, that the SPACE has to be sent to the VIO window and not to STDIN. This should cover the majority of cases.

It's not the Vio Window that's waiting for input, it's the process that's running in the shell that's running in the client area of the Vio Frame Window that's waiting for a key press. Even if you subclassed the client window to process a message, that's still not going to send a keypress to the process running in the shell that's running in the client window, because that process will be using KbdXXXXXX functions to read a physical key press, it knows nothing about messages.

What might work is to switch focus to the Vio Frame Window before showing the prompt in the editor window using WinSetFocus, that way the Vio Window gets the keypress and not the editor window. However, if that works at all, it's incredibly fragile, a simple alt-tab by the user breaks it.

I still can't see a use case for this feature in a text editor. I can see the user wanting to be able to transform text using complex pipelines and batch processing, but why would they be processing text through either another full screen interactive text editor or a pager as part of that process? The output is going to be appearing in the text editor, so I don't need a paginator when I can scroll backwards and forwards through the output text and I'm in a full screen interactive text editor, why would I want to run another one as a subprocess? It seems to be a misfeature that nobody would use except by accident, and that's what the interrupt or break signals  are for.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 08, 2020, 09:35:26 pm
At the moment, my program is already very useful. But I do not want that a user always has to think of if my program fits his purpose or not. So I have to add basic VIO switch functionality. The user often does not know which commands are VIO relevant and which not, even I have to try some programs to see what happens. So the behaviour of my editor should be usable, it does not need to be perfect. It is clear for me that we discuss perhaps 3% of the normal commandline usage, because the majority of commands is stdout-based only and the problems we discuss here do not appear.

Another simple solution of the MORE problem is to create a warning message in the editor. Because the more command depends on the size (line count) of the VIO window, MORE is not a good solution. I have got already a silder where you can reduce the output speed. The leftmost value could generate a pause after one visible page has been written with stdout output. This solution would be completely in the editor. Another option is that this behaviour can be set for one single command by using MORE. So my editor interpretates the MORE command and the rest of the command gets transformed to the command without MORE which is sent to cmd.exe.

An important question for me is if MORE is the only program which requires single keystrokes without creating VIO output before. Does anyone know a second program? If noone knows a second program, this internal editor solution would be perfect.

Anyway, when I publish the first version, everyone can report specific problems.

Title: Re: Observing if a program uses vio calls
Post by: Dave Yeo on March 08, 2020, 10:55:48 pm
Well there's various more replacements such as less, which is recommended to use instead of more.
What about if a program pipes output, something like "dir | tee dir.log"? or input, "\usr\bin\patch.exe < foo.patch"?
Title: Re: Observing if a program uses vio calls
Post by: Laurence Pithie on March 09, 2020, 12:02:25 am
There are a host of pagers, including variants which pop up PM windows. Another common scenario is piping text through troff into a postscript viewer.  It's not just Vio programs, most *nix derived utilities if you don't supply a filename will read from stdin by default, and in that scenario if you forget to provide a filename you will not get a prompt returned from the helper program to indicate in the PM program that it's waiting for input.
Title: Re: Observing if a program uses vio calls
Post by: Andreas Schnellbacher on March 09, 2020, 01:34:23 am
What about if a program pipes output, something like "dir | tee dir.log"? or input, "\usr\bin\patch.exe < foo.patch"?
Both already work with EPM. The current discussion is about VIO apps that don't use STDOUT or STDERROR but use VIO functions to write something in the console. Unfortunately there exist a few.

BTW: I don't see the necessity to support these cases, even against the background that there exist a few famous .exes and REXX .cmds that use VIO functions to write to the console or to place the cursor.
Title: Re: Observing if a program uses vio calls
Post by: Martin Vieregg on March 09, 2020, 09:23:36 am
When the "|" command is used to send data to another stdin/out program, I see no problems with my program. Only more and less and its derivates  (which names are also common?) can make problems. But these programs are helpers in a small text window where the output runs into Nirvana. This is the main functionality of my frontend to scroll back without limit. So these helpers are obsolete. I only have to find a useful way how to handle it. Like described above, I could support "| more" in the editor to enable the page-keypress functionality in my editor. In the ME editor when typing
[C:\] dir /s | me in
the default behaviour is not to scroll while output gets received, and this behaviour should be an option in my commandline frontend. My program has already a LED which glows red while the command is compiled and green when a prompt is shown. With ESC, you can easy jump to the prompt.

Our discussion here is very helpful for me. It helps me sorting my own ideas and it prevents me from programming something which does not fit the real purpose of my program.

I will publish a first version in 1-2 weeks. I have decided to name it MeShell.