OS2 World Community Forum

WebSite Information => Article Discussions => Topic started by: Lars on January 26, 2018, 10:35:46 am

Title: AN changing USB driver stack to 32-bit
Post by: Lars on January 26, 2018, 10:35:46 am
AN is currently working on XHCI. Along with that there is a decision to make the USB driver stack (mostly) 32-bit.
The stated reason for this was to increase data throughput.
Today, the USBxHCD.SYS and USBD.SYS drivers have been changed. Along with that change came a change in driver interface between these drivers that is no longer publicly documented (The driver interface of the old 16-bit drivers is documented in a HTML document as part of the DDK and therefore available to anyone with a DDK license).

Currently, the client drivers (USBPRT.SYS, USBMSD.ADD, USBAUDIO.SYS etc.) have not been changed and the driver interface of USBD.SYS still allows the use of 16-bit client drivers with the new 32-bit components.

However, there are ideas to also change the driver interface of all client drivers and make them 32-bit also.
What that means in practice is that you will no longer be able to combine say USBMSD.ADD and USBAUDIO.SYS from my driver set with the AN driver set.

Since AN has only little interest in offering audio and video support in OS/2 (it's not their business use case) that effectively would mean that development of USBAUDIO.SYS would be terminated and also that ArcaOS is become more and more proprietary, effectively a new OS next to OS/2.

Please speak up now if you don't want that to happen. I have always provided free USB drivers to the community and Wim and me we have invested considerable effort in making USBAUDIO.SYS what it is today. I see AN's need to make money from ArcaOS but I also see that with an ever diminishing user base, this step will be the last nail in the coffin of OS/2.
While there might be technical reasons I think that changing the driver interface will lock out anybody else in developing a USB driver (think about Wim's USBECD.SYS driver for example) and the OS/2 community is already short on driver developers.

Lars

Title: Re: AN changing USB driver stack to 32-bit
Post by: Mike Kölling on January 26, 2018, 02:38:04 pm
Hi Lars,
This is bad news for me, because I need both paths of development from you and AN. On my TP X200 I can only run the AN branch of USB drivers successful (or the latest IBM version) together with Wims drivers for the built in camera. On my newer TP X250 only your drivers are working. I really hope that both branches will not divert to much.
Greetings from Seoul,
Mike
Title: Re: AN changing USB driver stack to 32-bit
Post by: Neil Waldhauer on January 26, 2018, 03:41:43 pm
I'd rather see Arca Noae publish every interface that you guys need to continue their work. I can't imagine that they want the OS/2 community to shrink.

Going to a more modern 32-bit model for USB is not a bad thing in itself. Why make it bad by not disclosing necessary technical details?
Title: Re: AN changing USB driver stack to 32-bit
Post by: Lars on January 26, 2018, 04:02:50 pm
I'd rather see Arca Noae publish every interface that you guys need to continue their work. I can't imagine that they want the OS/2 community to shrink.

Going to a more modern 32-bit model for USB is not a bad thing in itself. Why make it bad by not disclosing necessary technical details?

It's not so much the bitsize that worries me as such but a change in interface.
For example, even in the existing USBEHCD.SYS I had changed part of the code to work with linear addresses (therefore 32-bit) because structures to manage became larger than 64 kBytes. That was necessary to get isochronous transfers to work with USB 2.0. There are other places in the drivers where I thunk to 32-bit and back to 16-bit. But all that was transparent to the inter-driver interface (no change).
My worry is that if data structures are passed between drivers that all pointers within these structures or the addresses of these structures will change to 32-bit flat or that the layout of these structures changes. When that happens, the existing client drivers will not work any more.
Also, it will then become impossible for me to run a USB driver with my stack in order to debug it (which I do quite frequently if there is a problem with driver interaction).
Title: Re: AN changing USB driver stack to 32-bit
Post by: Martin Iturbide on January 26, 2018, 05:50:16 pm
Hi

If Arca Noae's reason not to open source or distribute their documentation is because of the IBM DDK license excuse, I recommend for them to do what Lar's did. Offering USB drivers source code as a close source, but collaborative project under the umbrella of an organization like Netlabs.

That is the way they can legally continue USB development to be collaborative if they are worried about the IBM DDK license.

Regards
Title: Re: AN changing USB driver stack to 32-bit
Post by: Andi B. on January 27, 2018, 04:49:39 pm
Quote
AN is currently working on XHCI. Along with that there is a decision to make the USB driver stack (mostly) 32-bit.
The stated reason for this was to increase data throughput.
That's not completely accurate. At least not in the way a user would understand 'increase in throughput'. Even David said he does not expect measurable transfer speed or throughput changes.

My measurements when accessing external harddisks gave only slight improvement. But we are still way behind other platforms. And you have to mix Davids drivers with Lars usbmsd.add for getting best throughput. You may see less cpu utilization especially on old hardware and much concurrent accesses. But I would not expect a noticeable effect on current systems. YMMV.

Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day. 
Title: Re: AN changing USB driver stack to 32-bit
Post by: Sergey Posokhov on January 27, 2018, 11:00:46 pm
Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day.
Somebody's got a star fever, isn't it?..  :P
Title: Re: AN changing USB driver stack to 32-bit
Post by: Lars on January 28, 2018, 10:18:34 am
Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day.

If what you say is true I sincerely hope that at least the interface will be published. Because if not, nobody will be able to write a custom USB driver for a specific USB device, for example. Again, think about Wim's USBECD.SYS driver.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Joop on January 28, 2018, 01:39:36 pm
Its very simple. If AN does do that and doesn't publish with the result of no working existing drivers and no way out in developing new, then for me its game over. I need my (usb) pen or I will get severe trouble with my hand. If AN chooses this path then it will be a sort term vision for AN. I don't believe that DDK license is valid today because IBM did quit with OS/2. There business is not harmed with it because they don't sell OS/2 anymore. Simple as that. If they are active with it it will be an other matter, but so far no noise from IBM about OS/2.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Andi B. on January 28, 2018, 05:43:45 pm
I think we should note that most of what we are talking here is speculation. When Lars says -
Quote
Please speak up now if you don't want that to happen.
I fully understand his feelings. But on the other hand I'm not aware of any official statement from AN about that all until now. So what about should we speak up?

I'm quarreling with myself if I should post in this thread again. Until now AFAIK only testers knows about the new USB stack. I'm still unsure if testers should twaddle about things they test. I think it's better to calm down and see how it goes. But this discussion has started so I should add that the new 32 bit test stack does still work with Lars usbmsd.add.

Has anyone tested this new stack and found out that it does not work anymore with any other current driver (usbaudio, Wims driver, ....)? If yes, which driver does not work anymore?

@Joop
Quote
I don't believe that DDK license is valid today because IBM did quit with OS/2. There business is not harmed with it because they don't sell OS/2 anymore.
IMHO you are wrong. The DDK license will not go away cause only we would like it that way. And IBM does still sell OS/2 licenses. Otherwise AN was not able to sell you ArcaOS.

Title: Re: AN changing USB driver stack to 32-bit
Post by: ivan on January 28, 2018, 06:04:42 pm
Andi, you write about testers but the problem is just what are they testing on?  Are there any testers testing on AMD Ryzen or Theradripper or the A8 or A10 or even any of the newer AMD series of processors and APUs?  I very much doubt it which is a great cause for concern (I would have thought that they would be looking for the widest range of equipment for testing rather than what appears to be just Intel systems and older AMD offerings).

I will admit that I have an interest because I will not have an Intel processor in any of my equipment even notebooks and the lack of testing does show up.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Doug Bissett on January 28, 2018, 07:27:41 pm
Quote
Are there any testers testing on AMD

Yes.

Quote
I very much doubt it which is a great cause for concern

Do not be concerned.

At the moment, the 32 bit USB drivers are being TESTED, to be sure that the concept works. It does. When the drivers are actually RELEASED, then it becomes your concern to test them, to be sure that they work for you (if they don't, you can report the problem, and AN will try to fix it). At that time, it is also up to Arca Noae to release the source (or not), which will NOT include the DDK, You will still need the DDK to be able to use the source. Since that has always been the case, there should be no concern (the concern would be that somebody illegally releases the DDK, and gets the whole project into trouble with IBM and the law). If someone cares to recreate the DDK, LEGALLY, I am sure that very few people would be able to make use of it anyway.

Quote
I will admit that I have an interest because I will not have an Intel processor in any of my equipment even notebooks and the lack of testing does show up.

What do you mean by that statement? I have preferred AMD, for many years now, and haven't seen any problems because of it. I do have machines that are Intel (donated to me by windows users who felt compelled to buy a new machine), but they certainly don't work any better, or worse, than AMD. You MAY run into troubles if the machine is UEFI without a functional Compatibility Support Module (CSM), or if they don't have a DVD drive, or USB 2.x capability (USB 3.x support is getting closer, and part of that was to convert the drivers to 32 bit).

For your information, I am TESTING with:
Asus A88XM-A motherboard (AMD, A6, UEFI), with 8 GB memory.
Asus M3A78-EM motherboard (AMD), with 5 GB memory.
Lenovo ThinkPad T510 (Intel), with 8 GB memory.
Lenovo ThinkPad L530 (Intel UEFI), with 8 GB memory.
IBM ThinkPad T43 (Intel), with 1.5 GB memory.
IBM ThinkPad A22e (Intel), with 256 MB memory.
DELL Dimension 4300 (Intel), with 1 GB memory.
DELL Optiplex GX-270 (Intel), with 3 GB memory.
Asus P4VP-MX motherboard (Intel), with 1 GB memory.
HP p6242f (Intel) with 8 GB memory, and a NVIDIA video adapter (NOT recommended).

Overall, the DELL and HP,  machines have problems with video, when I try to use a wide screen. The IBM ThinkPad A22e is, of course, almost useless because the memory is maxed out at 256 MB, but it does work well, as long as I stay with programs that will run without having to page. The DELL 4300 has a problem with Air Boot, so I need to use Boot Manager with it (it is NOT an Air Boot problem).

FTR: The P4VP-MX has a PCI SATA adapter, with a 2 TB SATA drive attached and a 1 Gbit NIC. I use it as an OS/2 (ArcaOS) based NAS box.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Andi B. on January 28, 2018, 08:46:10 pm
Quote
Andi, you write about testers but the problem is just what are they testing on?
Where is the problem here? I guess if testers were paid by AN or by any other individual or company they will test what they get paid for. Otherwise they will test what they are interested in and with hardware they have.

If you want better support for your individual hardware you can open tickets in the bugtracker. If you want even better support I guess AN is happy to take your money and will invest in better support for your specific hardware. But it's up to AN to decide on that.

Title: Re: AN changing USB driver stack to 32-bit
Post by: Martin Iturbide on January 29, 2018, 03:28:00 am
Hi

It seems that there still doubts about IBM's DDK license. You can read the license here (https://www.os2world.com/wiki/index.php/IBM_OS/2_Products_Licensing_Analysis#IBM_Device_Driver_Kit_-_2004), specially the part of the "1. Grant of License for the IBM Code".  The IBM DDK is "Abandonware (https://en.wikipedia.org/wiki/Abandonware)".

Wikipedia defines abandonware as:
Quote
Abandonware is a product, typically software, ignored by its owner and manufacturer, and for which no support is available. Although such software is usually still under copyright, the owner may not be tracking copyright violations.

In intellectual rights context, abandonware is a software (or hardware) special case of the general concept of orphan works.

It mean that it may be illegal to distribute the software, but the copyright holder will not be enforcing the copyright of it.  But if someone starts making millions with the IBM DDK, it may raise one IBM eyebrow, and they have all the right to sue this people/company for all the money they want. But the chance of someone making millions or enough money to generate IBM's interest is unlikely.

But let's also remember that the IBM DDK code was available for free on their web site. It required free of charge registration of the developer. The developer will require to agree to the IBM DDK license to get access to download the driver sample source code.

I still think that the legal way to override any problem with IBM on this case (in the unlikely case it happens) is to create a close source collaborative project on Netlabs. Netlabs as organization that agreed to the IBM DDK license and any member of the Netlabs community that want to access the project will need to agreed to that license to have access to the source code and the modifications. This is the way Lar's USB project works and how ACPI used to be on Netlabs. What I don't agree is that a company grabs the Netlab's source code (IBM DDK source code with modifications) and create derivative close source projects without sharing back to the original source and/or without providing any extra documentation and knowledge back to the community.

So, the IBM DDK license is no excuse to create close/non-collaborative drivers. Do not believe the illuminati brotherhood when they tell you "You have to be lawyer to understand the IBM DDK license"


Regards
Title: Re: AN changing USB driver stack to 32-bit
Post by: Neil Waldhauer on January 29, 2018, 04:31:42 am
Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day.

If what you say is true I sincerely hope that at least the interface will be published. Because if not, nobody will be able to write a custom USB driver for a specific USB device, for example. Again, think about Wim's USBECD.SYS driver.

Did you test the new 32-bit driver with USBECD.SYS? Maybe it still works?

Title: Re: AN changing USB driver stack to 32-bit
Post by: Lars on January 29, 2018, 10:19:44 am
Did you test the new 32-bit driver with USBECD.SYS? Maybe it still works?

I have not tested. Looking at the USBECD.SYS source code, it expects the "standard" interface to USBD.SYS to register itself with USBD.SYS. Then, it offers the "standard" IDC entry point for being called back by the HC drivers on request termination ("standard" = as defined in the HTML document that is part of the DDK).
That is, it behaves like any other class device driver. Currently, these interfaces have not been touched and therefore it should still work.

I am talking about if AN decides to change these interfaces and making them proprietary.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Greg Pringle on January 29, 2018, 02:35:06 pm
Am I missing something? What does a DDK license have to do with a new interface designed by AN. That by definition is an AN product. They would be free to disclose the interface.

Simply use a text editor and start listing the the header files and add some usage info.

What am I missing?
Title: Re: AN changing USB driver stack to 32-bit
Post by: Martin Iturbide on January 29, 2018, 07:32:51 pm
What am I missing?

Hi Greg.

On the past the IBM DDK license has been use as an excuse to do not share any kind of driver development or documentation on this platform.
I just want to kill the excuse :)

The real problem is that Lars and Wim will not be able to continue their developments on the new USB stack, if not information and/or source code is shared.

Quote
What does a DDK license have to do with a new interface designed by AN.
For what is known AN is using IBM DDK source code samples for theirs drivers. So, the drivers source code are in part ruled by the IBM DDK license.

Regards
Title: Re: AN changing USB driver stack to 32-bit
Post by: ivan on January 29, 2018, 11:51:11 pm
Let me drop something else in here.

In developing usb3 drivers are they going to include something to enable use of the ASMedia usb3/sata chips or is that impossible because non of the testers have systems using those chips?
Title: Re: AN changing USB driver stack to 32-bit
Post by: Dave Yeo on January 30, 2018, 12:32:08 am
Hi Ivan, I think there is a mis-conception of the testers group. It was originally formed to test ANPM and then re-purposed for beta testing other things including Arcaos. I assume that Arca Noae has a group of machines for development and initial testing and hopefully it is a wide range of machines and while the testers do test on even more machines, we're perhaps mainly putting the installer through more varied tests.
In the case of the new USB stack, to quote David,
Quote
This is what the next release of the USB stack will look like. Feel free to
use this software if you like. I am not asking for testing since this
software has been thoroughly tested and has passed the standard QA
testing. The major changes are:
...
And then mentioning the possibility of timing issues and such with other software.
There hasn't been any reports that I'm aware of of breakages.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Doug Bissett on January 30, 2018, 01:05:07 am
Quote
Did you test the new 32-bit driver with USBECD.SYS? Maybe it still works?

I use USBECD.SYS with the mAPCUPS package. It never occurred to me that it might not work. It does work. Why would anyone think otherwise?

Quote
On the past the IBM DDK license has been use as an excuse to do not share any kind of driver development or documentation on this platform.
I just want to kill the excuse

This is silly. The DDK is required to use what AN developed, whether the driver is 32 bit, or not. AN cannot include the DDK in the code that they release to the public, but I expect the rest of it to be released, WHEN THE DRIVER IS FINISHED, and not before. Currently, the driver is under development, and testing. It is NOT available to the general population, yet. AN will also release their binary driver, using the official channels (Subscription, or ArcaOS license). Anybody else who wants it are on their own. If they have the DDK, and understand how to build the package, it should be possible, after AN finds the time to update the public repositories, and that may take a while, because those guys are busting their butts to get ArcaOS 5.0.2 ready for their users. It also makes sense to delay releasing the code, to make it more desirable to have a subscription, or buy ArcaOS (did I say that?).

Quote
Am I missing something? What does a DDK license have to do with a new interface designed by AN.

It is NOT a NEW interface. As I understand it, all that was done, was to replace all 16 bit code with 32 bit code. Previously, the (available) DRV32 package was incorporated to make it easier to use the DDK. The DDK is still copyright, and requires a license to use it. The results can be freely distributed, as the owner sees fit. AN sells support, and the service to build the binary, for those who buy a subscription.

Quote
In developing usb3 drivers are they going to include something to enable use of the ASMedia usb3/sata chips or is that impossible because non of the testers have systems using those chips?

In fact, I suspect (unfounded speculation, because I don't know), that the USB 3 part of the driver will be lifted from Linux, or BSD, so if Linux supports your chip, so will ArcaOS. FWIW, I have three different USB 3 adapters to test with, when the time comes. If you post the PCI output, that describes the parts, somebody may be able to say that they do have one that is similar (or not, as they choose).
Title: Re: AN changing USB driver stack to 32-bit
Post by: Martin Iturbide on January 30, 2018, 02:01:40 am
Quote
On the past the IBM DDK license has been use as an excuse to do not share any kind of driver development or documentation on this platform.
I just want to kill the excuse

This is silly. The DDK is required to use what AN developed, whether the driver is 32 bit, or not. AN cannot include the DDK in the code that they release to the public, but I expect the rest of it to be released, WHEN THE DRIVER IS FINISHED, and not before. Currently, the driver is under development, and testing. It is NOT available to the general population, yet. AN will also release their binary driver, using the official channels (Subscription, or ArcaOS license). Anybody else who wants it are on their own. If they have the DDK, and understand how to build the package, it should be possible, after AN finds the time to update the public repositories, and that may take a while, because those guys are busting their butts to get ArcaOS 5.0.2 ready for their users. It also makes sense to delay releasing the code, to make it more desirable to have a subscription, or buy ArcaOS (did I say that?).

Hi Doug, you seem to miss my earlier post.

Do you have hope that AN release the USB 3.0 source code to the USB Netlabs repository eventually. It may not be right away, but have you hear from them any intention to do that in the future ?

Regards
Title: Re: AN changing USB driver stack to 32-bit
Post by: Doug Bissett on January 30, 2018, 06:39:02 am
Quote
Do you have hope that AN release the USB 3.0 source code to the USB Netlabs repository eventually.

I can't speak for AN, but as long as it can be done without breaking copyright laws, or licensing agreements, i expect that it will be released, when they are ready to do so, and that will likely be AFTER they finish the USB 3 support. I do know that everyone involved with AN are working long hours getting ArcaOS 5.0.2 ready to go. I doubt if USB 3 will be ready to go for that release, but it is a big step in the right direction, to get the drivers converted to 32 bit.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Greg Pringle on January 30, 2018, 04:19:36 pm
It would be nice to have USB 3
A new ASUS H270M PLUS motherboard did not have any working USB. The backward compatible part did not work.
I added a StarTech USB 2.0 card with a Via chip set and all is well. I then had to convert to case wiring to allow USB 2.0 on the front panel.

The joys of OS/2.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Joop on February 02, 2018, 06:34:59 pm
I really don't know what this is all about. I read this in "Input/Output Device Driver Reference for OS/2";

COPYRIGHT LICENSE: This publication contains printed sample application programs in source language, which illustrate OS/2 programming techniques. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the OS/2 application programming interface.

Each copy of any portion of these sample programs or any derivative work, which is distributed to others, must include a copyright notice as follows: "(C) (your company name) (year).  All rights reserved."

(C) Copyright International Business Machines Corporation 1996. All rights reserved.
Note to U.S. Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

I don't read that you may not distribute it, that you may not modify it or that you may not copy it. This one comes from OS/2 Developer Connection Device Driver Kit, Version 4 Edition (June 1996).
Title: Re: AN changing USB driver stack to 32-bit
Post by: Doug Bissett on February 02, 2018, 08:23:59 pm
Quote
I don't read that you may not distribute it, that you may not modify it or that you may not copy it. This one comes from OS/2 Developer Connection Device Driver Kit, Version 4 Edition (June 1996).

The problem is not with distributing the final result. The problem is, that you cannot buy, rent, or sell, the license to use the DDK without IBM being involved. IBM no longer sells it, so your only option is to buy the license from somebody who already has it. Then,the seller needs to go through some legal procedure to convince IBM that they have destroyed all copies that they have, and get the new licensee registered as being licensed. Since IBM has no way to do that (and they can veto the new license anyway), you can't do it. Without the DDK, you cannot build the driver(s), so only those who have a license can build drivers that use the DDK.

Whether IBM would care, is another matter, but that is something that Arca Noae cannot take a chance on. If IBM goes after them, for some reason, they are out of business, and OS/2 will likely come to an end.
Title: Re: AN changing USB driver stack to 32-bit
Post by: Martin Iturbide on February 02, 2018, 10:04:57 pm
I really don't know what this is all about. ...

Hi Joop.

1.- I see the problem from the Open source perspective. The license on the "1. Grant of License for the IBM Code" section it says:

Quote
In addition, IBM grants to you the non-exclusive, non-assignable, non-transferable right, under the applicable IBM copyrights, to reproduce and distribute, in object code form only, the IBM Code and/or the permitted derivative work thereof, but only in conjunction with and as part of the OS/2 Device Driver and only if you:

a) do not make any statements to the effect or which imply that the OS/2 Device Driver is "certified" by IBM or that its performance is guaranteed by IBM and

b) agree to indemnify, hold harmless and defend IBM and its subsidiaries and their suppliers from and against any and all claims, legal proceedings, liabilities, damages, costs and expenses, including attorney's fees, arising out of or in connection with your distribution of the IBM Code and/or the OS/2 Device Driver.

You must reproduce any copyright notice(s) on each copy, or partial copy, of the IBM Code. If you redistribute any of the AFM and/or PPD files you must include the following copyright notice: "Copyright 1988, 1989 Adobe Systems Incorporated. All Rights Reserved".


Uhm, reading it complete gives me a different context. I need to read it all again.


2.- -About using the IBM DDK, since IBM DDK was available with a free of charge registration on the IBM site, I think you can still use it legally while you comply to the license agreement that IBM provided. At least until IBM publish an announcement letter saying that is taking back the right of use on the IBM DDK.

Regards