T30 Touch stall

Hi,

We do have some randomly problems with T30 modules and the touch. Periodically the touch fail on some of them. No response when the display is touched.
Everything work fine. Application, Ethernet, keyboard, USB, mouse etc.

Doing a warmboot or a coldboot, from the update tool, doesn’t solve it.
Only a complete power off will make the touch work again.

Software and image

We are running latest WEC2013 build version 2.0.
T30 1.1E (not 100% sure.)

Hardware and configuration:

We use KOE TX20D33VM2BPA 8" with a 4 wire resistive touch.
The 4 touch wire has each a 1 nF capacitor to ground.

On the AD1 we have a 330 Ohm pull down and a 10nF capacitor to ground.
On the AD3 we have VCAP voltage from backup capacitors. (2x 25 F). Voltage will maximum be 2.55V.

We also have a RTC DS1339U using the RTCSync driver. (Registry setting 2)

They both (RTC and Touch chip) use the I2C bus. Could it be a lock up there?

Any suggestions where to look?

Thomas

Nothing help to get on this one?

@TJO: Do you get any output on the Debug Output? On a reset are the touch and I2C resetted as well or not?

Hi,

I havn’t enabled debug output yet. I will do that as soon as possible.

The touch is part of the display, so there are no reset signal to that. The RTC chip we use have no reset signal. They are first reset if we power off. I do not about the internal I2C internal on the T30 module, if they get reset or not.

Thomas

Hi,

Finnaly I have succeded get a T30 to stall with debug output enabled.
It seems to be an issue with I2C RTC Sync driver?

This is just a litle part og the log, but it is full witht he same text:

STMPE811:NvError I2C arbitration failedTps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6236xRailSetVoltage(): Error in writing the Voltage register

STMPE811:NvError I2C arbitration failedTps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6236xRailSetVoltage(): Error in writing the Voltage register

STMPE811:NvError I2C arbitration failedTps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Tps6236xRailSetVoltage(): Error in writing the Voltage register
Tps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6236xRailSetVoltage(): Error in writing the Voltage register

STMPE811:NvError I2C arbitration failedTps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Tps6591xVDDSetVoltage() Error in reading the Voltage register
Tps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

HI,

I finally got the Touch stall to happen with debug output enabled.
It seems to be related to the RTC Sync driver and I2C.

What happens this time, the touch stopped working. Suddenly the T30 rebooted and everything was ok

This is a little part from the log, but it keeps repeating it self:

STMPE811:NvError I2C arbitration failedTps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register
STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006
STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

@TJO: The T30 part should be reseted. Please let me know if you see anything special on the debug port, may we can provide you further help then.

When touch stop working the debug out is full og this:

Err in I2c transfer: Controller Status 0x00000006
NvOdmPmuI2cWrite8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6236xRailSetVoltage(): Error in writing the Voltage register
AudioCodecRead: Error in the i2c communication 0x00045005
Tps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6236xRailSetVoltage(): Error in writing the Voltage register
AudioCodecWrite: Error in the i2c communication 0x00045005
Tps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register
AudioCodecWrite: Error in the i2c communication 0x00045005
Tps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register
AudioCodecWrite: Error in the i2c communication 0x00045005
Tps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedTps6236xPmuSetVoltage:The failure in enabling voltage rail 1
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6591xVDDSetVoltage() Error in reading the Voltage register

STMPE811:NvError I2C arbitration failedTps6591xPmuSetVoltage:The failure in enabling voltage rail 4
Err in I2c transfer: Controller Status 0x00000006

STMPE811:NvError I2C arbitration failedErr in I2c transfer: Controller Status 0x00000004
NvOdmPmuI2cRead8() Failed: status 0x7
Err in I2c transfer: Controller Status 0x00000006
Tps6236xRailSetVoltage(): Error in writing the Voltage register

@TJO: The problem seems to be, that the touch blocks the I2C line for to long. Internally the Power Chip that switches voltages depending on the CPU state is attached on the same I2C lines. If the CPU needs an other voltage, but the power chip can not deliver this, this could eventually result in a freeze / reboot of the device. I don’t think it is the RTCSync that makes trouble it is on a different RTC channel. Do you have any chance understand if your touch keeps sending touch events, that may could lead to this situation? Is there any special situation which triggers the behaviour? Did you made any changes on the touch registry settings: Resistive Touch Screen on WinCE | Toradex Developer Center

Hi Samuel,

I just had another one that had a Touch stall in production. I attached a mouse, but it was running very slowly. After 5-10 minutes it ran fast again, but still no touch.
I did a cold boot from the update tool, but the screen just kept being off (I enable backlight gpio with the display driver setup in registry)
I than did a power cycle and the touch worked again.

I dont guess my touch send any touch event. It is a basic 4 wire resistive touch, using the default touch controller in the image. If it did I would assume the touch would behave strangely and not be able to touch anything?

No special event triggering the event. This makes it difficult for me to debug because it do not happen very often.

I have not made any changes to the touch registry settings

Thomas

Hi,

This morning I had another one that had a complete system freeze (Touch, keybord, display stall)

It had the same in the debug log as above, but also this shown below. It keep repeating it self. I guess the exception in NK.exe is why the system has a complete freeze.

Any way to check if I have a lot of touch event?

BVA=407d12cc, dwInfo = 00000007
R0=00000001 R1=6000009f R2=6000001f R3=a0141108
R4=a28177d0 R5=b011fc4c R6=00000000 R7=ffffc800
R8=00000000 R9=00000001 R10=ffffc888 R11=ffffc824
R12=ffffc800 SP=b011fb40 Lr=8008b6c8 PC=8008b6c8, Psr=2000001f
Exception ‘Data Abort’(4) Thread-Id=00bb000a(pth=fffdd5dc) PC=80bd523c BVA=407d12cc, dwInfo = 00000007
R0=a28177d0 R1=407d12cc R2=00000000 R3=fffdd7ac
R4=fffdd800 R5=00000000 R6=a28177d0 R7=00000001
R8=00000000 R9=00000080 R10=a0004000 R11=37ff91ff
R12=83083214 SP=fffdd7a8 Lr=00000001 PC=80bd523c, Psr=20000013
Exception ‘Data Abort’ (0x4): Thread-Id=00bb000a(pth=a28177d0), Proc-Id=00400002(pprc=830835e0) ‘NK.EXE’, VM-active=042d0002(pprc=a288b568) ‘explorer.exe’
PC=80bd523c(???+0x80bd523c) RA=00000001(???+0x00000001) SP=fffdd7a8, BVA=407d12cc
Exception ‘Data Abort’(4) Thread-Id=02110002(pth=a28393c4) PC=8008b6c8 BVA=e59fc160, dwInfo = 00000005
R0=00000001 R1=00000000 R2=2000001f R3=00000001
R4=a28393c4 R5=b084fcd8 R6=00000000 R7=fffdb800
R8=00000000 R9=00000001 R10=ffffc888 R11=ffffc824
R12=fffdd800 SP=b084fbcc Lr=8008b6c8 PC=8008b6c8, Psr=2000001f
Exception ‘Data Abort’(4) Thread-Id=02110002(pth=fffdb5f4) PC=80081950 BVA=e59fc160, dwInfo = 00000005
R0=00000001 R1=00000001 R2=e59fc0ac R3=800675d4
R4=fffdb800 R5=8308301c R6=00000000 R7=00000001
R8=00000000 R9=00000001 R10=ffffc888 R11=ffffc824
R12=83083214 SP=fffdb7c0 Lr=80082db0 PC=80081950, Psr=20000193
Exception ‘Data Abort’ (0x4): Thread-Id=02110002(pth=a28393c4), Proc-Id=00400002(pprc=830835e0) ‘NK.EXE’, VM-active=042d0002(pprc=a288b568) ‘explorer.exe’
PC=80081950(kernel.dll+0x0000e950) RA=80082db0(kernel.dll+0x0000fdb0) SP=fffdb7c0, BVA=e59fc160
Exception ‘Data Abort’(4) Thread-Id=00b5000a(pth=a2817148) PC=8008b6c8 BVA=8b8a365b, dwInfo = 00000001
R0=00000001 R1=6000009f R2=6000001f R3=a0141108
R4=a2817148 R5=afbffc4c R6=00000000 R7=ffffc800
R8=00000000 R9=00000001 R10=ffffc888 R11=ffffc824
R12=ffffc800 SP=afbffb40 Lr=8008b6c8 PC=8008b6c8, Psr=2000001f
Exception ‘Data Abort’(4) Thread-Id=00b5000a(pth=fffd95e8) PC=80082150 BVA=8b8a365b, dwInfo = 00000001
R0=83085c20 R1=00000002 R2=00000000 R3=00000000
R4=83085c20 R5=00000000 R6=00000002 R7=00000001
R8=00000000 R9=00000000 R10=00c02140 R11=34bfbe7f
R12=8b8a3657 SP=fffd97b4 Lr=8008b53c PC=80082150, Psr=40000013
Exception ‘Data Abort’ (0x4): Thread-Id=00b5000a(pth=a2817148), Proc-Id=00400002(pprc=830835e0) ‘NK.EXE’, VM-active=042d0002(pprc=a288b568) 'explorer.exe

Hi,

I just have to add that this is seen with WEC7 latest release.
The original first post in this thread was with a WEC2013 image, but we have seen a lot with WEC7. All other posting with debug output log is with the latest WEC7 image!

Sorry abouyt the confusion…

@TJO: In the meantime we wrote some kind of a stress test tool for to stress the internal I2C bus. We got the case we have been able to crash the I2C as well as only the touch was not working any longer. We currently do not understand what happens here but we will further investigate.

Do you do some other communication on this internal I2C bus on your side, that may could increase the load on the bus?

Do you use audio? If not, please disable audio once (See also Disable drivers | Toradex Developer Center) and check if this makes any differnce.

Hi,

Thanks for the input.
No, we do not use the I2C bus for other stuff. The only load we have is the Touch and what ever the T30 module use internally (RTC, PMU, Audio etc)

I will try disable audio and see if that helps.
But in the future we need the audio.

Question: Does this also apply to the WEC2013 image?

Thomas

@TJO: This also apply to the WEC2013 image, yes. Please let me know your findings about the audio setting.

Yes, I will start using the modules with no audio, and monitor over the next week or two to see if the stall still happens.

Hi,

DIsabling audio do not help.
Just got one i production that has stall with the audio disabled.

Any new findings at you end?

Thomas

@TJO: There was no progress on our side. We will plan some resources to investigate this more deeply. We will provide you some updates as soon as we have a fix for this.

@TJO: After we firth thought we found kind of the issue, we unfortunately found out the issue we found is not related you yours. We made a mistake in the testing tool and wrote to I2C Address 0x78 which caused an bus communication issue.

Currently we further investigate what could be wrong as well. One other thing that could be possible: Did you check if you see this issue only on one module or also on others? If it is only one, it could be an RMA (for example a resistor missing).

Once you see the issue, May could you measure the value of the following two pins as marked in the attached graphics? What values do you see?

Thanks for the feedback.

It do happen with different modules and it is not isolated to one specific module. It do happen once in a while during production and I have seen it 3-4 times on different units during long time testing. There where 4 stalls last week in production on 4 different units.

Is it is possible to see if the Touch driver do generate a lot of traffic? Debug function or other way to hook onto the touch driver? Just to see if I can find anything here in my end.

I don’t think It would be possible to measure the values whn it happen. It only happen once in a while, so the change for it to happen on non-assembled unit on a desk is very unlikely.

Would it be possible to schedule some telephone meeting with you guys? To talk about how we can solve this issue, further action and time frame/schedule?

Thomas