I2C not working after waking up from suspend by touchscreen interrupt

CristianM · October 8, 2021, 2:43pm

I trying to wake up the iMX6ULL from suspend by pressing the touchscreen. I am using the display ETM070080BDH6, that integrates the touch controller Focaltech FT5406 - datasheet, Linux driver. This is my device tree binding:

polytouch: edt-ft5x06@38 {
		compatible = "edt,edt-ft5406", "edt,edt-ft5x06";
		reg = <0x38>;
		pinctrl-names = "default";
		interrupt-parent = <&gpio5>;
		interrupts = <4 IRQ_TYPE_EDGE_FALLING>; // SODIMM 107
		reset-gpios = <&gpio1 10 GPIO_ACTIVE_LOW>; // SODIMM 106
		wake-gpios = <&gpio5 10 GPIO_ACTIVE_HIGH>;
		status = "okay";
		wakeup-source;
        interrupt-names = "irq", "wakeup";
	};

The SoM wakes up from suspend when I touch the screen, but afterwards something happens to the I2C bus because I cannot access any device on the bus, nor the kernel. This is the dmesg output when entering and exiting suspend mode:

[ 2085.832414] PM: suspend entry (deep)
[ 2085.854194] Filesystems sync: 0.021 seconds
[ 2085.889088] Freezing user space processes ... (elapsed 0.004 seconds) done.
[ 2085.893549] OOM killer disabled.
[ 2085.893560] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 2085.904076] mwifiex_sdio mmc1:0001:1: info: successfully disconnected from 28:ee:52:33:87:70: reason code 3
[ 2085.906676] mwifiex_sdio mmc1:0001:1: None of the WOWLAN triggers enabled
[ 2086.011753] PM: suspend devices took 0.110 seconds
[ 2086.015956] Disabling non-boot CPUs ...
[ 2086.018637] imx-i2c 21a0000.i2c: Unbalanced pm_runtime_enable!
[ 2086.026894] edt_ft5x06 0-0038: Unable to fetch data, error: -13
[ 2086.039403] edt_ft5x06 0-0038: Unable to fetch data, error: -13
[ 2086.052873] edt_ft5x06 0-0038: Unable to fetch data, error: -13
[ 2086.071180] edt_ft5x06 0-0038: Unable to fetch data, error: -13
[ 2086.083979] edt_ft5x06 0-0038: Unable to fetch data, error: -13
[ 2086.195372] rtc-pcf8563 0-0051: pcf8563_read_block_data: read error
[ 2086.264789] PM: resume devices took 0.250 seconds
[ 2086.264811] OOM killer enabled.
[ 2086.264818] Restarting tasks ... done.
[ 2086.287795] PM: suspend exit

If I wake up the SoM by pressing a button (connected to a GPIO configured as a wakeup source), the I2C bus does not have any problems. It seems there is a problem with the edt-ft5x06 driver that causes the imx-i2c 21a0000.i2c: Unbalanced pm_runtime_enable! error on the I2C interface when resuming/returning from suspend, but I don’t have a clue what it is.

Can someone provide some help in debugging and hopefully resolving this issue?

Colibri iMX6ULL 512MB WB IT
Custom carrier board
Linux TDX X11 5.2.0-devel-20210226152038+build.0 (dunfell)

saijanani.tx · October 28, 2021, 9:16am

Hello @CristianM ,

I saw from the other thread of yours that the issue could be in the Atmel driver. So, I was wondering if you still need any assistance on the Toradex side, or is this thread closed?

Best Regards,
Janani

CristianM · October 28, 2021, 10:57am

@saijanani.tx Hi. Yes, I still need assistance because this question refers to the ETM070080BDH6 display, which integrates the touch controller Focaltech FT5406.

CristianM · November 16, 2021, 2:39pm

Does anybody have a clue how to resolve this issue?

DaveM · November 16, 2021, 5:32pm

You might want to follow that advice in the other thread and start diffing the driver code with the current branches. Lots of power management changes running around there…

CristianM · November 16, 2021, 6:07pm

@DaveM I have analyzed the driver code, but unfortunately I do not have sufficient experience to identify which function call is causing this behavior, nor do I know how to compile a modified driver.

saijanani.tx · November 17, 2021, 4:51pm

Hello @CristianM ,

Thank you for your patience in this regard. Meanwhile, I have tried to use the EDT driver with the Toradex display and it seems to work. We do not readily have the display you mentioned at our office to test it quickly. Also, this display doesn’t seem to be present in the list of supported & tested displays

But still, if you think you need expert support on this topic and dig more deeper, I would like to suggest you check out the Support hours page.

I hope this somehow helps.

BR, Janani

CristianM · November 17, 2021, 5:43pm

Hopefully you can get the display I mentioned and test it.

saijanani.tx · November 18, 2021, 3:58pm

Hello @CristianM ,

As the display you are mentioning is not in the list of the supported displays yet and as we currently don’t have enough resources at the moment, I, unfortunately, cannot promise you that this will certainly happen. But, if it is a pressing issue at your end that needs immediate attention, I kindly suggest you to consider the Support hours.

Thank you for your understanding!

Best Regards,
Janani

smagrian · November 19, 2021, 11:45am

@CristianM Have you tried to change the driver you are using to the version from the Linux kernel repository? That one

I have encountered the same problem recently - after updating to the newest kernel/Yocto/BSP, when exiting suspend mode, the whole i2c bus would get completely stuck. Any operation on it would return permission denied error, the same you are seeing I believe (error code = 13)
I checked recent changes in the driver code and for us, implementing changes from these two commits helped:

one
two

But in your controller’s case, the range of changes in power management code is much bigger so I think I would start with trying to exchange the whole file.

CristianM · November 19, 2021, 12:44pm

@smagrian Thank you for your response. I have tried to use the driver, but I could not compile it into the kernel in Yocto. I would be very grateful if you can tell me how to do this or how you managed to do it.

smagrian · November 19, 2021, 5:00pm

@CristianM Hmm I am not an expert in Yocto but I would try something like this:

create a .patch file containing the differences between your current driver and the actual code you want to use,
create a .bbappend file that would append to the original linux-toradex recipe. I do not know how your build system is organized but if you are using some custom meta layer, the file should probably be located in:
layers/<your-meta-layer>/recipes-kernel/linux/linux-toradex_%.bbappend
create linux-toradex catalogue in that directory, and place that .patch file created earlier into that catalogue. So the file tree would look something like this:

layers/<your-meta-layer>/recipes-kernel/linux/
| 
|--linux-toradex_%.bbappend
|--linux-toradex/
    |
    |--patch-to-driver.patch

your .bbappend file should then look somewhat like this:

FILESEXTRAPATHS_prepend := "${THISDIR}/linux-toradex:"

SRC_URI += "\ 
	file://patch-to-driver.patch \
	"

Using this approach, i think bitbake should apply your .patch file to the code, before attempting to build the driver.

CristianM · November 22, 2021, 2:11pm

@smagrian Hi. I followed your advice and created the folder structure indicated. I created the patch file using the following command:

diff edt-ft5x06.c edt-ft5x06_new.c > edt-ft5x06.patch

where edt-ft5x06_new.c contains the updated driver. I have attached the edt-ft5x06.patch (14.4 KB)
file.

This is my .bbappend file (named linux-toradex_5.4.91.bbappend):

FILESEXTRAPATHS_prepend := "${THISDIR}/linux-toradex:"

SRC_URI += "\ 
	file://edt-ft5x06.patch \
	"

The problem is that I get the following error:

bitbake linux-toradex_5.4.91.bbappend
Loading cache: 100% |####################################################################################| Time: 0:00:00
Loaded 3827 entries from dependency cache.
WARNING: No recipes available for:
  /home/cristi/oe-core/build/../layers/meta-customer/recipes-kernel/linux/linux-toradex_5.4.91.bbappend
ERROR: Nothing PROVIDES 'linux-toradex_5.4.91.bbappend'

Please do you have any suggestions?

rafael.tx · November 22, 2021, 2:33pm

Hi @CristianM,
your bbappend file must have the exact same name as the kernel recipe that your are appending to, with the .bb replaced by .bbappend. Do you have a kernel recipe file with the name linux-toradex_5.4.91.bb?

veskola · November 22, 2021, 3:06pm

Hi,

I have similar problem with iMX8MM and custom carrier board. We have four I2C chips in same bus and after suspend most of the I2C chips stops to respond, but not every time. I have figured out the first chip, in my case, SN65DSI83 bridge chip, respond to I2C writes after suspend but then the bus is left somehow in busy state and it results -13 error for ever.

I already modified SN65DSI83 driver, gpiod_set_value instead of gpiod_set_value_cansleep method. It had some influence but the problems is randomly still there. Probably that just change some timings but did not fix the root cause.

I will also try to upgrade edt-ft5x06 driver, actually I already have patch file for that for other reason. Anyhow i’m a bit skeptic the root cause is there.

I’m also wondering what that Unbalanced pm_runtime_enable message means, it is not seen if resume from suspend is ok.

[ 316.459576] CPU3 is up
[ 316.481324] imx-i2c 30a50000.i2c: Unbalanced pm_runtime_enable!
[ 316.488779] imx-drm soc@0:bus@32c00000:display-subsystem: sn65dsi83_bridge_mode_set: mode: 800*480@29700
[ 316.501001] imx-drm soc@0:bus@32c00000:display-subsystem: sn65dsi83_bridge_enable
[ 316.508507] sn65dsi83 3-002c: sn65dsi83_brg_setup
[ 316.513220] sn65dsi83 3-002c: sn65dsi83_brg_power_on
[ 316.549102] sn65dsi83 3-002c: DSI clock [ 89100000 ] Hz
[ 316.554341] sn65dsi83 3-002c: GeoMetry [ 800 x 480 ] Hz
[ 316.560863] sn65dsi83 3-002c: client 0x0000000040a6c3b1
[ 316.576225] sn65dsi83 3-002c: sn65dsi83_brg_start_stream
// there SN65DSI83 I2C went OK
[ 316.789370] sn65dsi83 3-002c: client 0x0000000040a6c3b1
[ 316.794898] sn65dsi83 3-002c: CHA (0xe5) = 0x01
[ 316.837131] caam 30900000.crypto: registering rng-caam
[ 316.935883] OOM killer enabled.
[ 316.940791] Restarting tasks … done.
[ 316.942867] power_supply bq256xx-charger: driver failed to report `manufacturer’ property: -13
// Further all I2C in this 03 bus is blocked.

[edit: I have now updated edt-ft5x06.c from Linux 5.16-rc2, lot of code changes but same problem remain. It seems somehow bridge driver use after suspend get I2C blocked]

CristianM · November 22, 2021, 4:30pm

@rafael.tx Thank you for the response, but how do I find the recipe that created/compiled the driver/module edt-ft5x06.c?

veskola · November 22, 2021, 6:35pm

You should have tdx/layers/something/something/recipes/kernel/linux/linux-toradex_%.bbappend which add .patch file. It will modify tdx/layers/meta-toradex-bsp-common/recipes-kernel/linux/linux-toradex-mainline_5.4.bb recipe.

Typically whole kernel and linux image is created for with bitbake tdx-reference-multimedia-image command

CristianM · November 29, 2021, 7:16pm

@veskola @smagrian Thank you for the advice. I managed to compile the patch using the filename linux-toradex_%.bbappend. Then, I updated the kernel by copying the new kernel (zImage) to the device, run the command ubiupdatevol /dev/ubi0_0 zImage and rebooted the device.

The problem is that after the new kernel boots, my kernel-managed I2C devices do not appear as managed (UU) when scanning the I2C bus with i2cdetect. Also, the touchscreen does not work.

It seems that the patch had a negative effect. I installed the previous kernel to get everything working again.

I have attached my folder for applying the patch (found in my custom Yocto layer).

Any suggestions?

veskola · November 30, 2021, 6:44am

Patch file seems to be identical with me, only difference is format of patch file (I’m using git format-patch --signoff HEAD~1 command) but it does not explain why I2C is totally broken now. I’ pathing on top of Linux version 5.4.154-5.4.0-devel+git.9f04e0d3d560 kernel

An other difference is in .bbappend file, I’m using SRC_URI_append_verdin-imx8mm-MyMachineType because I have custom machine in use which is defined in local.conf. I assume your version is ok, when using default machine type.

I’m building whole image with Yocto, and not using zImage update method, unfortunately do not have experience of that. Kernel boot log probably indicate what is happening for I2C chips/bus when kernel parse device tree during boot. There should be something like:
[ 2.067990] i2c i2c-3: IMX I2C adapter registered
…
[ 2.379201] input: EVERVISION-FT5726NEi as /devices/platform/soc@0/soc@0:bus@30800000/30a50000.i2c/i2c-3/3-0038/input/input1

In my case there is seen Evervision-ft5726nei which is edt-ft5x06 compatible touch chip. Same device tree was ok, with or without patch.

CristianM · December 3, 2021, 2:03pm

@veskola My Linux version is 5.2.0-devel-20210226152038+build.0 (dunfell).

I use the kernel (zImage) update method, because it is easier and faster to test than deploying the entire image every time on the device to see if the patch worked.

I have attached the dmesg output before and after applying the patch. From what I can tell, maybe the patch does not work with my version of Linux.

What do you think?