Author Topic: AN changing USB driver stack to 32-bit  (Read 28208 times)

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
AN changing USB driver stack to 32-bit
« on: January 26, 2018, 10:35:46 am »
AN is currently working on XHCI. Along with that there is a decision to make the USB driver stack (mostly) 32-bit.
The stated reason for this was to increase data throughput.
Today, the USBxHCD.SYS and USBD.SYS drivers have been changed. Along with that change came a change in driver interface between these drivers that is no longer publicly documented (The driver interface of the old 16-bit drivers is documented in a HTML document as part of the DDK and therefore available to anyone with a DDK license).

Currently, the client drivers (USBPRT.SYS, USBMSD.ADD, USBAUDIO.SYS etc.) have not been changed and the driver interface of USBD.SYS still allows the use of 16-bit client drivers with the new 32-bit components.

However, there are ideas to also change the driver interface of all client drivers and make them 32-bit also.
What that means in practice is that you will no longer be able to combine say USBMSD.ADD and USBAUDIO.SYS from my driver set with the AN driver set.

Since AN has only little interest in offering audio and video support in OS/2 (it's not their business use case) that effectively would mean that development of USBAUDIO.SYS would be terminated and also that ArcaOS is become more and more proprietary, effectively a new OS next to OS/2.

Please speak up now if you don't want that to happen. I have always provided free USB drivers to the community and Wim and me we have invested considerable effort in making USBAUDIO.SYS what it is today. I see AN's need to make money from ArcaOS but I also see that with an ever diminishing user base, this step will be the last nail in the coffin of OS/2.
While there might be technical reasons I think that changing the driver interface will lock out anybody else in developing a USB driver (think about Wim's USBECD.SYS driver for example) and the OS/2 community is already short on driver developers.

Lars


Mike Kölling

  • Full Member
  • ***
  • Posts: 130
  • Karma: +6/-0
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #1 on: January 26, 2018, 02:38:04 pm »
Hi Lars,
This is bad news for me, because I need both paths of development from you and AN. On my TP X200 I can only run the AN branch of USB drivers successful (or the latest IBM version) together with Wims drivers for the built in camera. On my newer TP X250 only your drivers are working. I really hope that both branches will not divert to much.
Greetings from Seoul,
Mike

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1024
  • Karma: +24/-0
    • View Profile
    • Blonde Guy
Re: AN changing USB driver stack to 32-bit
« Reply #2 on: January 26, 2018, 03:41:43 pm »
I'd rather see Arca Noae publish every interface that you guys need to continue their work. I can't imagine that they want the OS/2 community to shrink.

Going to a more modern 32-bit model for USB is not a bad thing in itself. Why make it bad by not disclosing necessary technical details?
Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #3 on: January 26, 2018, 04:02:50 pm »
I'd rather see Arca Noae publish every interface that you guys need to continue their work. I can't imagine that they want the OS/2 community to shrink.

Going to a more modern 32-bit model for USB is not a bad thing in itself. Why make it bad by not disclosing necessary technical details?

It's not so much the bitsize that worries me as such but a change in interface.
For example, even in the existing USBEHCD.SYS I had changed part of the code to work with linear addresses (therefore 32-bit) because structures to manage became larger than 64 kBytes. That was necessary to get isochronous transfers to work with USB 2.0. There are other places in the drivers where I thunk to 32-bit and back to 16-bit. But all that was transparent to the inter-driver interface (no change).
My worry is that if data structures are passed between drivers that all pointers within these structures or the addresses of these structures will change to 32-bit flat or that the layout of these structures changes. When that happens, the existing client drivers will not work any more.
Also, it will then become impossible for me to run a USB driver with my stack in order to debug it (which I do quite frequently if there is a problem with driver interaction).

Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4710
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: AN changing USB driver stack to 32-bit
« Reply #4 on: January 26, 2018, 05:50:16 pm »
Hi

If Arca Noae's reason not to open source or distribute their documentation is because of the IBM DDK license excuse, I recommend for them to do what Lar's did. Offering USB drivers source code as a close source, but collaborative project under the umbrella of an organization like Netlabs.

That is the way they can legally continue USB development to be collaborative if they are worried about the IBM DDK license.

Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #5 on: January 27, 2018, 04:49:39 pm »
Quote
AN is currently working on XHCI. Along with that there is a decision to make the USB driver stack (mostly) 32-bit.
The stated reason for this was to increase data throughput.
That's not completely accurate. At least not in the way a user would understand 'increase in throughput'. Even David said he does not expect measurable transfer speed or throughput changes.

My measurements when accessing external harddisks gave only slight improvement. But we are still way behind other platforms. And you have to mix Davids drivers with Lars usbmsd.add for getting best throughput. You may see less cpu utilization especially on old hardware and much concurrent accesses. But I would not expect a noticeable effect on current systems. YMMV.

Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day. 

Sergey Posokhov

  • Full Member
  • ***
  • Posts: 169
  • Karma: +8/-6
    • View Profile
    • OS/2 API Research
Re: AN changing USB driver stack to 32-bit
« Reply #6 on: January 27, 2018, 11:00:46 pm »
Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day.
Somebody's got a star fever, isn't it?..  :P

Lars

  • Hero Member
  • *****
  • Posts: 1271
  • Karma: +65/-0
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #7 on: January 28, 2018, 10:18:34 am »
Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day.

If what you say is true I sincerely hope that at least the interface will be published. Because if not, nobody will be able to write a custom USB driver for a specific USB device, for example. Again, think about Wim's USBECD.SYS driver.

Joop

  • Hero Member
  • *****
  • Posts: 633
  • Karma: +5/-0
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #8 on: January 28, 2018, 01:39:36 pm »
Its very simple. If AN does do that and doesn't publish with the result of no working existing drivers and no way out in developing new, then for me its game over. I need my (usb) pen or I will get severe trouble with my hand. If AN chooses this path then it will be a sort term vision for AN. I don't believe that DDK license is valid today because IBM did quit with OS/2. There business is not harmed with it because they don't sell OS/2 anymore. Simple as that. If they are active with it it will be an other matter, but so far no noise from IBM about OS/2.
« Last Edit: January 28, 2018, 02:42:27 pm by Joop »

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #9 on: January 28, 2018, 05:43:45 pm »
I think we should note that most of what we are talking here is speculation. When Lars says -
Quote
Please speak up now if you don't want that to happen.
I fully understand his feelings. But on the other hand I'm not aware of any official statement from AN about that all until now. So what about should we speak up?

I'm quarreling with myself if I should post in this thread again. Until now AFAIK only testers knows about the new USB stack. I'm still unsure if testers should twaddle about things they test. I think it's better to calm down and see how it goes. But this discussion has started so I should add that the new 32 bit test stack does still work with Lars usbmsd.add.

Has anyone tested this new stack and found out that it does not work anymore with any other current driver (usbaudio, Wims driver, ....)? If yes, which driver does not work anymore?

@Joop
Quote
I don't believe that DDK license is valid today because IBM did quit with OS/2. There business is not harmed with it because they don't sell OS/2 anymore.
IMHO you are wrong. The DDK license will not go away cause only we would like it that way. And IBM does still sell OS/2 licenses. Otherwise AN was not able to sell you ArcaOS.


ivan

  • Hero Member
  • *****
  • Posts: 1557
  • Karma: +17/-0
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #10 on: January 28, 2018, 06:04:42 pm »
Andi, you write about testers but the problem is just what are they testing on?  Are there any testers testing on AMD Ryzen or Theradripper or the A8 or A10 or even any of the newer AMD series of processors and APUs?  I very much doubt it which is a great cause for concern (I would have thought that they would be looking for the widest range of equipment for testing rather than what appears to be just Intel systems and older AMD offerings).

I will admit that I have an interest because I will not have an Intel processor in any of my equipment even notebooks and the lack of testing does show up.

Doug Bissett

  • Hero Member
  • *****
  • Posts: 1593
  • Karma: +4/-2
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #11 on: January 28, 2018, 07:27:41 pm »
Quote
Are there any testers testing on AMD

Yes.

Quote
I very much doubt it which is a great cause for concern

Do not be concerned.

At the moment, the 32 bit USB drivers are being TESTED, to be sure that the concept works. It does. When the drivers are actually RELEASED, then it becomes your concern to test them, to be sure that they work for you (if they don't, you can report the problem, and AN will try to fix it). At that time, it is also up to Arca Noae to release the source (or not), which will NOT include the DDK, You will still need the DDK to be able to use the source. Since that has always been the case, there should be no concern (the concern would be that somebody illegally releases the DDK, and gets the whole project into trouble with IBM and the law). If someone cares to recreate the DDK, LEGALLY, I am sure that very few people would be able to make use of it anyway.

Quote
I will admit that I have an interest because I will not have an Intel processor in any of my equipment even notebooks and the lack of testing does show up.

What do you mean by that statement? I have preferred AMD, for many years now, and haven't seen any problems because of it. I do have machines that are Intel (donated to me by windows users who felt compelled to buy a new machine), but they certainly don't work any better, or worse, than AMD. You MAY run into troubles if the machine is UEFI without a functional Compatibility Support Module (CSM), or if they don't have a DVD drive, or USB 2.x capability (USB 3.x support is getting closer, and part of that was to convert the drivers to 32 bit).

For your information, I am TESTING with:
Asus A88XM-A motherboard (AMD, A6, UEFI), with 8 GB memory.
Asus M3A78-EM motherboard (AMD), with 5 GB memory.
Lenovo ThinkPad T510 (Intel), with 8 GB memory.
Lenovo ThinkPad L530 (Intel UEFI), with 8 GB memory.
IBM ThinkPad T43 (Intel), with 1.5 GB memory.
IBM ThinkPad A22e (Intel), with 256 MB memory.
DELL Dimension 4300 (Intel), with 1 GB memory.
DELL Optiplex GX-270 (Intel), with 3 GB memory.
Asus P4VP-MX motherboard (Intel), with 1 GB memory.
HP p6242f (Intel) with 8 GB memory, and a NVIDIA video adapter (NOT recommended).

Overall, the DELL and HP,  machines have problems with video, when I try to use a wide screen. The IBM ThinkPad A22e is, of course, almost useless because the memory is maxed out at 256 MB, but it does work well, as long as I stay with programs that will run without having to page. The DELL 4300 has a problem with Air Boot, so I need to use Boot Manager with it (it is NOT an Air Boot problem).

FTR: The P4VP-MX has a PCI SATA adapter, with a 2 TB SATA drive attached and a 1 Gbit NIC. I use it as an OS/2 (ArcaOS) based NAS box.

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: AN changing USB driver stack to 32-bit
« Reply #12 on: January 28, 2018, 08:46:10 pm »
Quote
Andi, you write about testers but the problem is just what are they testing on?
Where is the problem here? I guess if testers were paid by AN or by any other individual or company they will test what they get paid for. Otherwise they will test what they are interested in and with hardware they have.

If you want better support for your individual hardware you can open tickets in the bugtracker. If you want even better support I guess AN is happy to take your money and will invest in better support for your specific hardware. But it's up to AN to decide on that.


Martin Iturbide

  • OS2World NewsMaster
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4710
  • Karma: +41/-1
  • Your Friend Wil Declares...
    • View Profile
    • Martin's Personal Blog
Re: AN changing USB driver stack to 32-bit
« Reply #13 on: January 29, 2018, 03:28:00 am »
Hi

It seems that there still doubts about IBM's DDK license. You can read the license here, specially the part of the "1. Grant of License for the IBM Code".  The IBM DDK is "Abandonware".

Wikipedia defines abandonware as:
Quote
Abandonware is a product, typically software, ignored by its owner and manufacturer, and for which no support is available. Although such software is usually still under copyright, the owner may not be tracking copyright violations.

In intellectual rights context, abandonware is a software (or hardware) special case of the general concept of orphan works.

It mean that it may be illegal to distribute the software, but the copyright holder will not be enforcing the copyright of it.  But if someone starts making millions with the IBM DDK, it may raise one IBM eyebrow, and they have all the right to sue this people/company for all the money they want. But the chance of someone making millions or enough money to generate IBM's interest is unlikely.

But let's also remember that the IBM DDK code was available for free on their web site. It required free of charge registration of the developer. The developer will require to agree to the IBM DDK license to get access to download the driver sample source code.

I still think that the legal way to override any problem with IBM on this case (in the unlikely case it happens) is to create a close source collaborative project on Netlabs. Netlabs as organization that agreed to the IBM DDK license and any member of the Netlabs community that want to access the project will need to agreed to that license to have access to the source code and the modifications. This is the way Lar's USB project works and how ACPI used to be on Netlabs. What I don't agree is that a company grabs the Netlab's source code (IBM DDK source code with modifications) and create derivative close source projects without sharing back to the original source and/or without providing any extra documentation and knowledge back to the community.

So, the IBM DDK license is no excuse to create close/non-collaborative drivers. Do not believe the illuminati brotherhood when they tell you "You have to be lawyer to understand the IBM DDK license"


Regards
Martin Iturbide
OS2World NewsMaster
... just share the dream.

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1024
  • Karma: +24/-0
    • View Profile
    • Blonde Guy
Re: AN changing USB driver stack to 32-bit
« Reply #14 on: January 29, 2018, 04:31:42 am »
Guessing the change was made to make future development easier. For AN. Not for the rest of the world. Except David make his changes and his DDK publicly available some day.

If what you say is true I sincerely hope that at least the interface will be published. Because if not, nobody will be able to write a custom USB driver for a specific USB device, for example. Again, think about Wim's USBECD.SYS driver.

Did you test the new 32-bit driver with USBECD.SYS? Maybe it still works?

Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com