Author Topic: Use of webcam  (Read 78149 times)

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: Use of webcam
« Reply #45 on: June 02, 2020, 12:40:24 am »
I have read some of the ffmpeg documentation and rfc1341(mime).

Section 20.15 mpjpeg - MJPEG encapsulated in multi-part MIME demuxer. 
Suppose the webcam streams mjpg video through "\PIPE\Webcam\Video".
Would the following be correct to play that mjpg video?

"ffplay -f mpjpeg \PIPE\Webcam\Video"
"ffplay -f mpjpeg Strict_mime_boundary 0 \PIPE\Webcam\Video"
"ffplay -f mpjpeg Strict_mime_boundary 1 \PIPE\Webcam\Video"

What are the exact mime headers to be used?

Section 20.16 rawvideo - Raw video demuxer.
Suppose the webcam streams yuyv video through "\PIPE\Webcam\Video".
Would the following be correct to play that yuyv video?

"ffplay -f rawvideo -video_size 640x480 \PIPE\Webcam\Video"
"ffplay -f rawvideo -pixel_format yuv420p -video_size 640x480 -framerate 10 \PIPE\Webcam\Video"

The above assumes that the original mjpg/yuyv video from the webcam will be streamed.
An alternative could be to stream rgb24/mjpg snapshots from the application window?

For ffplay yuv, options are
rawvideo demuxer AVOptions:
  -video_size        <image_size> .D....... set frame size
  -pixel_format      <string>     .D....... set pixel format (default "yuv420p")
  -framerate         <video_rate> .D....... set frame rate (default "25")

For mjpeg, you have:
mjpeg demuxer AVOptions:
  -framerate         <video_rate> .D.......  (default "25")
mjpeg_2000 demuxer AVOptions:
  -framerate         <video_rate> .D.......  (default "25")

and mpjpeg using mime type "multipart/x-mixed-replace"

From you above example, it is supposed the webcam sends frames into a named pipe.
(If this could be done, it would be nice)
Under linux, ffmpeg as well ffplay are able to use specified hw input device like webcam but this is currently not implemented under os/2 (code not ported) 

Or uvcscope sending mjpeg (play video + send all mjpeg or only send mjpeg) to ffmpeg or ffplay (both using parm " f mjpeg -i - "  (reading stdin)
e.g.  uvvscope /dxxxx:xxxx:xxxx - | ffplay -f mjpeg -i -     (or   uvvscope /dxxxx:xxxx:xxxx - | ffmpeg -f mjpeg -i - record.avi   
But audio is missing out of uvcscope...

I think that Dave could help here to progress.
Regards
Rémy

 







Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: Use of webcam
« Reply #46 on: June 02, 2020, 04:20:16 am »
As far as I know, a command like
Code: [Select]
"ffplay -f mpjpeg \PIPE\Webcam\Video"
doesn't work on our command line. In theory you could use mkfifo to create FFmpeg friendly pipes, though I've never tried and don't know it mkfifo has been properly ported to OS/2.
Code: [Select]
[Y:\]mkfifo --help
Usage: mkfifo [OPTION]... NAME...
Create named pipes (FIFOs) with the given NAMEs.

Mandatory arguments to long options are mandatory for short options too.
  -m, --mode=MODE    set file permission bits to MODE, not a=rw - umask
  -Z                   set the SELinux security context to default type
      --context[=CTX]  like -Z, or if CTX is specified then set the SELinux
                         or SMACK security context to CTX
      --help     display this help and exit
      --version  output version information and exit

There's some examples here for concatenating, https://ffmpeg.org/faq.html#Concatenating-using-raw-audio-and-video

I'd guess if you can concatenate audio and video, you should be able to mux it together but I'm far from an expert at using FFmpeg, just building it.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Use of webcam
« Reply #47 on: June 02, 2020, 11:32:32 am »
Quote
  Interface 2: Alt=0 nEP=0 Class=1(Audio) SubClass=1 Protocol=0 StringIx=0
  Interface 3: Alt=0 nEP=0 Class=1(Audio) SubClass=2 Protocol=0 StringIx=0
  Interface 3: Alt=1 nEP=1 Class=1(Audio) SubClass=2 Protocol=0 StringIx=0
  Interface 3: Alt=2 nEP=1 Class=1(Audio) SubClass=2 Protocol=0 StringIx=0
  Interface 3: Alt=3 nEP=1 Class=1(Audio) SubClass=2 Protocol=0 StringIx=0
  Interface 3: Alt=4 nEP=1 Class=1(Audio) SubClass=2 Protocol=0 StringIx=0

The information on audio is rather meager i.e. even endpoint information is missing.
Fortunately I have lsusb reports from both c250 and c270 webcams and the audio descriptors appear to be identical.
The inbuild microphone may deliver 16-bit mono at 16000/ 24000/32000/48000 Hz.

I have experimented in the past to obtain audio from the c250 webcam and it appears to be possible.
The big question is what to do with the audio data once obtained. Should it go to the default audio device?
The truth is that I am not really an application programmer, so I want this to be as simple as it can be.

I also tried to have usbaudio.sys recognize the c250 microphone but I gave up on that.

Wim, can you send me the lsusb report for the C250 ? I suspect that there is some feature unit for volume/mute on the USB "data out" (microphone) path that needs to be cranked up (or a multiplexer unit set to the correct input or some such). Currently, USBAUDIO.SYS only controls the feature unit for volume/mute on the USB "data in" (speaker) path.

You can even try to write a REXX script to crank up that feature unit. You sure know how to do that once you find the corresponding feature unit in the descriptor info ...
« Last Edit: June 02, 2020, 11:35:52 am by Lars »

Wim Brul

  • Sr. Member
  • ****
  • Posts: 295
  • Karma: +25/-0
    • View Profile
    • Wim's home page
Re: Use of webcam
« Reply #48 on: June 02, 2020, 11:35:40 am »
Hi Rémy,

Quote
Under linux, ffmpeg as well ffplay are able to use specified hw input device like webcam but this is currently not implemented under os/2 (code not ported)

Perhaps 2 hardware devices are then being used (one for audio and one for video)? Would that be dll's (like video4linux.dll and alsa.dll)?

Quote
Or uvcscope sending mjpeg (play video + send all mjpeg or only send mjpeg) to ffmpeg or ffplay (both using parm " f mjpeg -i - "  (reading stdin)
e.g.  uvvscope /dxxxx:xxxx:xxxx - | ffplay -f mjpeg -i -     (or   uvvscope /dxxxx:xxxx:xxxx - | ffmpeg -f mjpeg -i - record.avi   

What do you mean by "play video + send all mjpeg or only send mjpeg" ? Would an application without video display (like my weblook.exe) do?

Quote
But audio is missing out of uvcscope...

Yes. To be done of course. But how? I looked into your avxcat.pdf and learned that audio and video are handled as separate inputs.
When both audio and video must be delivered then working with named pipes (one for audio and one for video)  would do I guess.
The webcam can only be accessed by one application, so how would I send both both audio and video through stdout?

Best regards,
Wim.

Wim Brul

  • Sr. Member
  • ****
  • Posts: 295
  • Karma: +25/-0
    • View Profile
    • Wim's home page
Re: Use of webcam
« Reply #49 on: June 02, 2020, 12:24:27 pm »
I also tried to have usbaudio.sys recognize the c250 microphone but I gave up on that.

Wim, can you send me the lsusb report for the C250 ? I suspect that there is some feature unit for volume/mute on the USB "data out" (microphone) path that needs to be cranked up (or a multiplexer unit set to the correct input or some such). Currently, USBAUDIO.SYS only controls the feature unit for volume/mute on the USB "data in" (speaker) path.

O.K. I have attached the lsusb reports for both the c250 and c270.
The problem (in fact feature) with usbaudio.sys is that it does not recognize the webcam as an audio device. It is a so-called miscellaneous device.
Once I fixed that. It appears that only the microphone has descriptors and usbaudio.sys expects an audio output interface as well. It ran into traps.

Quote
You can even try to write a REXX script to crank up that feature unit. You sure know how to do that once you find the corresponding feature unit in the descriptor info ...

Yes I do. The question remains should usbaudio.sys support the webcam microphone. I doubt it.

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Use of webcam
« Reply #50 on: June 02, 2020, 02:00:59 pm »
Hi Wim,

does USBAUDIO.SYS also fail to find the microphone on the C-270 ?
For the C-250 and also for the C-270 I can see AUDIO CONTROL and also one or more AUDIO STREAMING interfaces and that's ok.
But there are also association descriptors. Maybe usbaudio.sys fails to properly find the control and streaming descriptors because it gets confused by the association descriptor.

Certainly, USBAUDIO.SYS should not trap if it finds a "record only" USB audio device. I have a USB microphone (only MIC). Let's see what that does to USBAUDIO.SYS.

I would say that USBAUDIO.SYS (and USBAUD2.SYS) should support everything that has a AUDIO CONTROL and AUDIO STREAMING interface (descriptor). It has to serve all devices of that class, even if there is more to the device than just audio.

The video part would then be handled by a USBVIDEO.SYS driver but you have decided to go a different route via USBCALLS/USBRESMG which is also ok.

Wim Brul

  • Sr. Member
  • ****
  • Posts: 295
  • Karma: +25/-0
    • View Profile
    • Wim's home page
Re: Use of webcam
« Reply #51 on: June 02, 2020, 03:52:14 pm »
Hi Lars,

Hi Wim,

does USBAUDIO.SYS also fail to find the microphone on the C-270 ?

I cannot try because I don't have the C270. But the C270 is also a miscellaneous device so usbaudio.sys will also ignore it.   

Quote
For the C-250 and also for the C-270 I can see AUDIO CONTROL and also one or more AUDIO STREAMING interfaces and that's ok.
But there are also association descriptors. Maybe usbaudio.sys fails to properly find the control and streaming descriptors because it gets confused by the association descriptor.

Certainly, USBAUDIO.SYS should not trap if it finds a "record only" USB audio device. I have a USB microphone (only MIC). Let's see what that does to USBAUDIO.SYS.

I must say that I did a dirty hack to accept the webcam and to find the feature_unit  descriptor. I seem to recall that the traps occur while attempting to record.

Quote
I would say that USBAUDIO.SYS (and USBAUD2.SYS) should support everything that has a AUDIO CONTROL and AUDIO STREAMING interface (descriptor). It has to serve all devices of that class, even if there is more to the device than just audio.

The video part would then be handled by a USBVIDEO.SYS driver but you have decided to go a different route via USBCALLS/USBRESMG which is also ok.

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: Use of webcam
« Reply #52 on: June 02, 2020, 03:54:03 pm »
Quote
Quote
Under linux, ffmpeg as well ffplay are able to use specified hw input device like webcam but this is currently not implemented under os/2 (code not ported)

Perhaps 2 hardware devices are then being used (one for audio and one for video)? Would that be dll's (like video4linux.dll and alsa.dll)?


To get video and audio, both interfaces have to be used and as I understand, 2 devices
Both formats video4linux and alsa are not available under ffmpeg yet (missing dlls)   

Quote
Quote
Or uvcscope sending mjpeg (play video + send all mjpeg or only send mjpeg) to ffmpeg or ffplay (both using parm " f mjpeg -i - "  (reading stdin)
e.g.  uvvscope /dxxxx:xxxx:xxxx - | ffplay -f mjpeg -i -     (or   uvvscope /dxxxx:xxxx:xxxx - | ffmpeg -f mjpeg -i - record.avi   

What do you mean by "play video + send all mjpeg or only send mjpeg" ? Would an application without video display (like my weblook.exe) do?


I just give a try to weblook and no, weblook seems to only open access to the webcam. additional code is needed like stream the webcam rawvideo to stdout (or pipe, or local udp port).
ffmpeg can have multiple entries and can handle video as well audio in one command line. Video from first input goes maps as e.g. 0:0 and audio from second input maps as e.g. 0:1 input final video

 ***

In following example, I mux video from video1 with audio from video2 (both mp4) into the pipe (converting into mpegts) and ffplay plays the pipe received video "TS format" (low quality due no other specified options)

having the video and audio as selected under ffmpeg.

ffmpeg -i video1.mp4 -i video2.mp4 -map 0:v:0 -map 1:a:0 -shortest -f mpegts pipe: | ffplay -i pipe:
same as:
ffmpeg -i video1.mp4 -i video2.mp4 -map 0:v:0 -map 1:a:0 -shortest -f mpegts - | ffplay -i -

test done with ffplay show me it only reads pipe:  (or - )

Same example using udp port:
ffmpeg -i video1.mp4 -i video2.mp4 -map 0:v:0 -map 1:a:0 -shortest -f mpegts udp://127.0.0.1:5000 | ffplay -i udp://0.0.0.0:5000

Using pipe or udp, the stream must be seekable (mp4 is no good, this is why I added mpegts)

video1.mp4 and video2.mp4 input can be file (like here) or pipe or udp

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: Use of webcam
« Reply #53 on: June 02, 2020, 04:41:55 pm »
Hi Lars,

Hi Wim,

does USBAUDIO.SYS also fail to find the microphone on the C-270 ?

I cannot try because I don't have the C270. But the C270 is also a miscellaneous device so usbaudio.sys will also ignore it.   

Quote
For the C-250 and also for the C-270 I can see AUDIO CONTROL and also one or more AUDIO STREAMING interfaces and that's ok.
But there are also association descriptors. Maybe usbaudio.sys fails to properly find the control and streaming descriptors because it gets confused by the association descriptor.

Certainly, USBAUDIO.SYS should not trap if it finds a "record only" USB audio device. I have a USB microphone (only MIC). Let's see what that does to USBAUDIO.SYS.

I must say that I did a dirty hack to accept the webcam and to find the feature_unit  descriptor. I seem to recall that the traps occur while attempting to record.

Quote
I would say that USBAUDIO.SYS (and USBAUD2.SYS) should support everything that has a AUDIO CONTROL and AUDIO STREAMING interface (descriptor). It has to serve all devices of that class, even if there is more to the device than just audio.

The video part would then be handled by a USBVIDEO.SYS driver but you have decided to go a different route via USBCALLS/USBRESMG which is also ok.
Hi Wim,
in the lsusb report I see a warning "Warning: Descriptor too short" which will break the descriptor chain and that is true for the C-250 and C-270. Maybe THAT is the reason why searching for the audio control and streaming interface descriptors fails because the OS/2 USB client drivers always search the chain by honouring the "bLength" field instead of using known fixed values.

Wim Brul

  • Sr. Member
  • ****
  • Posts: 295
  • Karma: +25/-0
    • View Profile
    • Wim's home page
Re: Use of webcam
« Reply #54 on: June 07, 2020, 07:38:32 pm »
Hi Rémy,

I did install ffmpeg and played around with it. I also modified weblook to provide an mjpeg video stream to stdout. That works now.
Code: [Select]
weblook | ffplay -f mjpeg -noinfbuf -framerate 30 -i -

I also modified weblook to provide an mjpeg video stream to http:\\127.0.0.1:14225\ and that works too (also with SeaMonkey).
Code: [Select]
ffplay -f mjpeg -noinfbuf -framerate 30 -i http://127.0.0.1:14225/

Right now it is a compile option to choose what video stream will be used. I intend to make it a run time option.

So far only video is provided. To combine audio with video I need to provide two separate streams. Doing that from one program mandates the use of named pipes. I tried to create named pipes with mkfifo but that did not work. To solve that I need to write a named pipe filter to stdout. When I provide audio from another program then it can just be streamed to stdout as well and that filter program would not be necessary. Unfortunately I cannot access the very same webcam from 2 different programs at the same time. So in that cause audio has to come from another source i.e. the multimedia subsystem via dart. Without webcam audio support in usbaudio.sys  it will have to be one of the multimedia devices attached to the system. When usbaudio.sys would be modified to recognize the microphone in the webcam then it becomes a problem when one is using an usb audio stick at the same time. Perhaps a difficult problem to solve. So for now I will try and write that filter program first.
« Last Edit: June 07, 2020, 07:46:46 pm by Wim Brul »

Rich Walsh

  • Sr. Member
  • ****
  • Posts: 331
  • Karma: +23/-0
  • ONU! (OS/2 is NOT Unix!)
    • View Profile
Re: Use of webcam
« Reply #55 on: June 07, 2020, 10:23:12 pm »
I cannot access the very same webcam from 2 different programs at the same time. So in that cause audio has to come from another source i.e. the multimedia subsystem via dart. Without webcam audio support in usbaudio.sys  it will have to be one of the multimedia devices attached to the system. When usbaudio.sys would be modified to recognize the microphone in the webcam then it becomes a problem when one is using an usb audio stick at the same time. Perhaps a difficult problem to solve. So for now I will try and write that filter program first.

I don't think you're looking at this properly because you have everything you'd need (except the code) to serve both audio and video.

Because you're using usbresmg/usbcalls, your program isn't just a server: it's also a device driver running at Ring 3. You own the device and can do with it as you wish. If it provides both audio and video interfaces, there's nothing stopping you from setting up simultaneous isochronous streams from each. Just as (I assume) you package a series of USB packet payloads into an MPEG frame, you'd do the same for the audio in the appropriate format.

What you do with the streams after you capture them is up to you...

Remy

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +9/-1
    • View Profile
Re: Use of webcam
« Reply #56 on: June 07, 2020, 10:36:31 pm »
Hi Rémy,

I did install ffmpeg and played around with it. I also modified weblook to provide an mjpeg video stream to stdout. That works now.
Code: [Select]
weblook | ffplay -f mjpeg -noinfbuf -framerate 30 -i -

I also modified weblook to provide an mjpeg video stream to http:\\127.0.0.1:14225\ and that works too (also with SeaMonkey).
Code: [Select]
ffplay -f mjpeg -noinfbuf -framerate 30 -i http://127.0.0.1:14225/

Right now it is a compile option to choose what video stream will be used. I intend to make it a run time option.

So far only video is provided. To combine audio with video I need to provide two separate streams. Doing that from one program mandates the use of named pipes. I tried to create named pipes with mkfifo but that did not work. To solve that I need to write a named pipe filter to stdout. When I provide audio from another program then it can just be streamed to stdout as well and that filter program would not be necessary. Unfortunately I cannot access the very same webcam from 2 different programs at the same time. So in that cause audio has to come from another source i.e. the multimedia subsystem via dart. Without webcam audio support in usbaudio.sys  it will have to be one of the multimedia devices attached to the system. When usbaudio.sys would be modified to recognize the microphone in the webcam then it becomes a problem when one is using an usb audio stick at the same time. Perhaps a difficult problem to solve. So for now I will try and write that filter program first.

Very nice and promising progress

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: Use of webcam
« Reply #57 on: June 08, 2020, 12:21:33 am »
Would libuvc, https://ken.tossell.net/libuvc/doc/ help with the video? Could try to build it.

Paul Smedley

  • Hero Member
  • *****
  • Posts: 2092
  • Karma: +159/-0
    • View Profile
Re: Use of webcam
« Reply #58 on: June 08, 2020, 01:55:35 am »
Hey Dave,

Would libuvc, https://ken.tossell.net/libuvc/doc/ help with the video? Could try to build it.

Assuming usbresmg/usbcalls now support ISO transfers, this should be pretty easy to build. In theory it would open up webcams to being used in other apps ported from Unix

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: Use of webcam
« Reply #59 on: June 08, 2020, 02:31:36 am »
Hi Paul, looking at it a bit harder, libuvc seems to require opencv https://opencv.org/, which doesn't look that easy to build. The Linux version requires GTK though it looks like on Windows it can use QT. Everything uses CMAKE by the looks of it.