SetCommState() fails on PC when connected to USB in CDC mode for iMX6

We’ve been using the USB connection to T20 modules in CDC mode (Virtual COM port) for a long time without any problems. I’ve just run into a problem with using the same code on the iMX6. What I’ve found is that I can open a connection from the PC to the USB port of the iMX6 using TeraTerm and communicate with my software running on the iMX6 with no problem. But if I try to open the port on the PC using .NET then it causes an exception with the message “A device attached to the system is not functioning.”

The symptoms are the same as those described in this old post:
CDC and .Net Framework Serial Ports - NETMF & Gadgeteer - GHI Electronics’ Forums

I found a post which suggested that the problem with the SerialPort.Open() method in .NET is that it calls the Win32 API SetCommState() which fails.

I wrote a short app to test the Win32 API on the PC and indeed SetCommState() fails when it is called with the same data returned by GetCommState(),

HANDLE hSerialPort = CreateFile(L"COM9", GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hSerialPort != INVALID_HANDLE_VALUE)
{
    // Get the port configuration
    DCB portDCB;
    portDCB.DCBlength = sizeof(DCB);
    GetCommState(hSerialPort, &portDCB);

    // Set the parameters without changing them
    BOOL result = SetCommState(hSerialPort, &portDCB);

    CloseHandle(hSerialPort);
}

The contents of the DCB structure returned by GetCommState() are:

DCBlength           28	
BaudRate            9600	
fBinary             1	
fParity             0
fOutxCtsFlow        0
fOutxDsrFlow        0
fDtrControl         0
fDsrSensitivity     0
fTXContinueOnXoff	0
fOutX               0
fInX                0
fErrorChar          0
fNull               0
fRtsControl         1
fAbortOnError       0
fDummy2             0
wReserved           0
XonLim              0
XoffLim             0
ByteSize            8
Parity              0
StopBits            0
XonChar             0
XoffChar            0
ErrorChar	0
EofChar	101
EvtChar	0
wReserved1	0

I think there might be something wrong with the USB serial driver in the iMX6 compared to the T20. For some of our products this is the primary way our customers control them so getting this to work with the iMX6 is important to us as we work to transition away from the now obsolete T20 modules.

Have you disable the ActiveSync on the module?

Yes, it is disabled using the registry. The port does work when used with some terminal applications on the PC, for example TeraTerm. I think that shows that ActiveSync is correctly disabled to allow the port to be used by my application on the Colibri module. The problem seems to be that the port does not behave in a way which is compatible with most applications on the PC, for example those that use the .NET SerialPort class.

Unfortunately, I have no experience with .NET. Could you reproduce this issue using C/C++ code?

Yes, it happens in C++ as well. The code snippet in the description above reproduces the problem by calling the CreateHandle(), GetCommState() and SetCommState() Win32 APIs from C/C++ on the PC. The symptom is that SetCommState() fails and returns FALSE.

The .NET SerialPort class is a wrapper for those Win32 API calls. So SerialPort.Open() calls the same Win32 API SetCommState() which fails, and that causes SerialPort.Open() to fail too.

Would you mind creating a simple WIN32 console project to demonstrate this issue and sharing it with us? Please double check if it works successfully with Colibri T20.

Thanks for engaging with this topic by the way @alex.tx
I switched back to a T20 installed in the same board and running the same firmware to double-check that it works OK with the T20 and doesn’t work with the iMX6 (It does work with the T20). I noticed a difference between the T20 and the iMX6 when I did this, I think this difference might be the root cause.

The difference is that the T20 USB connection appears on the PC as USB class 2, subclass FF, prot FF while the iMX6 USB connection appears as class 2, subclass 2, prot FF as shown below.

T20
iMX6

According to this page class 2 and subclass 2 specifies a modem, while class 2 and other values of subclass specifies a COM port. I think the problem might be that the iMX6 is advertising its USB connection as being a modem by mistake.
USB device class drivers included in Windows - Windows drivers | Microsoft Learn

This reminds me of an issue I reported a few years ago about the USB subclass not being correct for the RNDIS driver. The solution to that was that the Toradex team made a patched version of the relevant DLL which I installed on my Colibri module in order to correct the USB subclass number. That allowed for a quicker fix than waiting for the next release of the OS image and that type of fix could also work here.
RNDIS on Windows 10 to connect to CE7 running on T20 - Technical Support - Toradex Community

I also packaged my C++ code as a project for you. This simple code will prompt you for a COM port number and will then test the port and tell you if SetCommState() succeeds or not.
SetCommState.zip (33.0 KB)

I don’t know if it is helpful but here is what we have for iMX8M-Plus and works fine on Windows 11 for us.
image

Thanks for posting that @Fide, I notice that the iMX8 uses Prot 1 while the iMX6 uses Prot FF. I don’t know what that means but it might help Alex.

@alex.tx have you had a chance to try to reproduce the problem using the extra details and the C++ project which I shared with you?

@MikeS, To me if TeraTerm or Putty works fine, then that means you have an issue with your own coding. You need to question why you need to call SetCommState() for a virtual/non-physical port where BaudRate, Flowcontrols or any other settings are irrelevant. Also what is the source of the assumption that whatever you get with GetCommState() will be valid for SetCommState() as a parameter?

Did you try GetLastError() after GetCommState()? Maybe that function fails? If that is not the case what does GetLastError() return after SetCommState() ?

@Fide all good questions. TeraTerm does work but I have noticed that when I use TeraTerm to open the port for the first time after connecting the USB cable that there is a delay of a few seconds. During those few seconds TeraTerm becomes unresponsive and its Window title bar is changed to “TeraTerm has stopped responding”. In my own code the call to SetCommState() takes a few seconds to return so I think TeraTerm is probably calling SetCommState(), becoming unresponsive while SetCommState() runs and then TeraTerm carries on gracefully after SetCommState() fails.

My call to GetCommState() does succeed and returns TRUE, if I call GetLastError() then the result is zero for ERROR_SUCCESS. The call to SetCommState() fails and returns FALSE, and GetLastError() returns 31 which is ERROR_GEN_FAILURE with the description “A device attached to the system is not functioning”.

I believe it’s quite standard practice to call GetCommState() first to retrieve the existing settings, then change just the settings that need to be changed and pass the results back to SetCommState(). Microsoft’s documentation says, “The SetCommState function uses a DCB structure to specify the desired configuration. The GetCommState function returns the current configuration. To set only a few members of the DCB structure, you should modify a DCB structure that has been filled in by a call to GetCommState. This ensures that the other members of the DCB structure have appropriate values.”
SetCommState function (winbase.h) - Win32 apps | Microsoft Learn

You are quite right to point out that if I were writing my own application in C++ I could simply choose not to call SetCommState() when I know this is a virtual COM port because all the fields that are passed to SetCommState() are ignored in that case. The problem is that the application I deliver to my customer to control my product is written in C# and uses the .NET Framework. The Open() method in the .NET class SerialPort makes a call to SetCommState(), which causes Open() to fail, as described here:
CDC and .Net Framework Serial Ports - NETMF & Gadgeteer - GHI Electronics’ Forums

When researching this before I created this thread I found another page which described the same issue. That person went so far as to download the source for the SerialPort class from the Mono project, commented out the call to SetCommState() and verified that it worked. (I can’t find that link today).

So, I could solve this problem in my own .NET app by either hacking the SerialPort class, or by using P/Invoke to open the COM port using the underlying Win32 API. But that would only work for my app, it wouldn’t work for any of my customers who create their own .NET apps to control my product. It probably wouldn’t work from LabView either as it also uses .NET (Some of my customers and my colleagues use LabView to control my Toradex-based product as part of a larger system).

A long reply I know, but hopefully it helps to clarify why I am hoping to fix this.

1 Like

@alex.tx and @Fide I’ve made a big step forward on this topic, and realized I had created the main issue for myself. I remembered about this topic from 5 years ago:
Can we access the virtual COM port using COMx rather than \.\wceusbsh001? - Technical Support - Toradex Community

That was a similar discussion about USB class and subclass settings for the T20. Raja made a modified copy of the driver file for the T20 called serialusbfn.dll which uses USB class 2 and subclass 2. That driver allows the VCOM port on the T20 to be recognized and used Windows 10 & 11 without having to supply a .inf file, it’s plug and play.

What I realized today was that our firmware was still quietly installing this modified driver when it ran on the iMX6. That modified driver is clearly not 100% compatible with the iMX6. When I removed the modified driver and used the one which comes as part of the iMX6 image then the problem with opening the COM port and using SetCommState() goes away.

This is a big improvement, but it doesn’t completely solve the problem. The driver that is distributed with the iMX6 image uses USB class 2 and subclass 0xFF as shown below. Windows 10 & 11 don’t recognize that subclass and it appears in the device manager with an exclamation point until I tell Windows to load my custom and digitally signed .inf file. After that the COM port works very well.

image

Using subclass 0xFF was part of the discussion in that other topic from 5 years ago when Raja made a driver for me which uses subclass 2. The screenshot which @Fide posted earlie in this topic shows that the iMX8 uses subclass 2, which would explain why the COM port works perfectly with the iMX8. Can you make me a customized dll for the iMX6 which uses subclass 2? Making the COM port work in a plug and play way without the need to install a driver is a real benefit for our end users.

It seems that you have made some modifications in the registry related to USB serial settings. Could you please export the registry keys HKLM\System\CurrentControlSet\Control and HKLM\Drivers\USB\FunctionDrivers? for both cases: the working T20 and the problematic IMX6.

Please also try solution listed here - Can we access the virtual COM port using COMx rather than \\.\wceusbsh001? - #30 by raja.tx

@alex.tx I was using the driver from comment 18 in that thread. That’s the version which works on the T20 and almost works on the iMX6:
Can we access the virtual COM port using COMx rather than \.\wceusbsh001? - Technical Support - Toradex Community

I tried the driver from comment 30 on that thread as you suggested in your last comment:
Can we access the virtual COM port using COMx rather than \.\wceusbsh001? - Technical Support - Toradex Community

I found that the driver from comment 30 did not work at all on the iMX6. Windows doesn’t recognize that there is a device attached to the USB port, there is no beep when I connect the cable and nothing appears in the device manager. I notice that comment #30 refers to CE6, I am using CE7 could it be that the driver only supports CE6?

@alex.tx here are the two registry exports you asked for. I exported them while the default driver which is included with the iMX6 image was installed.
HKLM_SYSTEM_CurrentControlSet_Control.reg (4.2 KB)
HKLM_Drivers_USB_FunctionDrivers.reg (2.4 KB)

Hi @alex.tx, have you had a chance to look into this after I uploaded those files? We’re still hoping that you will be able to make a modified iMX6 DLL for us which uses subclass 2.

The serialusbfn.dll, which is provided by Microsoft, should be the same for both Tegra and iMX6 platforms. Therefore, it’s unusual to observe different behavior between them. Are you using the same version of WinCE in both cases?

Hi @MikeS,

Unfortunately i could not find the exact source of the patched version that Raja made a few years ago. But it looks like he only changed the bInterfaceSubClass field of the USB Descriptors.
Why this modified driver woul dnot work properly on iMX6 it’s a bit strange, but after reviewing the iMX6 BSP Source i saw that we did some patches on the serialusbfn.dll for the iMX6 that were not there on T20. It has something to do with the maximum transfer length for the USB_COMM_SET_LINE_CODING serial class request. It seems to have something to do with the GetCommState()
Not sure why this patch is not needed on T20, maybe the underlaying USB Function driver does more checks on T20.
As for you request to have a special version with bInterfaceSubClass = 2 i could create such a version, but maybe it’s more useful to add a registry entry to configure that.

Hi @germano.tx, sorry for taking such a long time to reply. I missed the notification for your update and was busy working on other issues. I am still very interested in this though. Using a registry setting to configure the USB subclass would work well, but that would mean asking you to create a new OS image. Using a modified dll has the advantage that I could use it with the existing OS image, which would give us a shorter development path. Or would you be able to create an image with the new registry key for me to test fairly easily?

Hi

On Windows could you try using \\.\COM9 instead of COM9? In C(++) don’t forget about special purpose of \ symbol, so “\\\\.\\COM9” in C. I’m adding those \\.\ since XP times, because it could work well without \\.\ on low numbered ports like COM2 COM3 and fail on something like COM9.

Edit: Community seems being C as well, had to edit to make all \ displayed :slight_smile: