U-boot cannot read SD card - is this expected to work?

matthijs · September 11, 2024, 5:06pm

Hi,

I’m working with a verdin-imx8mp and am having trouble getting u-boot to detect an SD card.

My goal is mostly to validate our custom carrierboard design, to see if the SD-card interface has the right pullup so Linux, Tezi and u-boot can read data from the SD card (and all other interfaces). The SD card already works as expected in Tezi and Linux, but somehow in U-boot it fails to detect the SD card:

Verdin iMX8MP # mmc info
MMC Device 0 not found
no mmc device at slot

Maybe I am using the wrong commands to test this, but it seems like u-boot does not detect the SD card. I’ve observed this on our custom CB, but also on the Toradex dev board, so I suspect I am doing something wrong, but I cannot quite figure out what.

Should I expect u-boot to be able to read an SD card? Does this need any commands other than the mmc info command (I also tried mmc rescan - same result).

To clarify, I am not asking to boot u-boot from SD card (I know this is not an option) - u-boot boots from eMMC as normal but should AFAICS be able to then be able to read files from SD card (and eventually also load an image from the SD card, but I’m stranded way before that).

The full U-boot output is below, which shows the second mmc device (eMMC) is detected as expected (which makes sense, since booting from eMMC works fine).


U-Boot SPL 2022.04-6.4.0+git.dc27426aa417 (Jan 01 1970 - 00:00:00 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Trying to boot from BOOTROM
Boot Stage: Primary boot
Find img info 0x&4802e000, size 888
Download 941056, Total size 941904
NOTICE:  BL31: v2.6(release):lf_v2.6-g3c1583ba0a
NOTICE:  BL31: Built : 00:00:00, Jan  1 1970


U-Boot 2022.04-6.4.0+git.dc27426aa417 (Jan 01 1970 - 00:00:00 +0000)

CPU:   i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 41C
Reset cause: POR
DRAM:  4 GiB
Core:  89 devices, 23 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex 0063 Verdin iMX8M Plus Quad 4GB IT V1.1A
Serial#: 07321726
Carrier: Toradex Verdin Development Board V1.1C, Serial# 10996012
SEC0:  RNG instantiated

 BuildInfo:
  - ATF 3c1583b

flash target is MMC:2
Net:   eth1: ethernet@30be0000, eth0: ethernet@30bf0000 [PRIME]
Fastboot: Normal
Normal Boot
Hit any key to stop autoboot:  0 
Verdin iMX8MP # mmc info
MMC Device 0 not found
no mmc device at slot 

Verdin iMX8MP # mmc list
FSL_SDHC: 1
FSL_SDHC: 2 (eMMC)
Verdin iMX8MP # mmc dev 2
switch to partitions #0, OK
mmc2(part 0) is current device
Verdin iMX8MP # mmc info
Device: FSL_SDHC
Manufacturer ID: 70
OEM: 0
Name: IX2932 
Bus Speed: 200000000
Mode: HS400ES (200MHz)
Rd Block Len: 512
MMC version 5.1
High Capacity: Yes
Capacity: 29.1 GiB
Bus Width: 8-bit DDR
Erase Group Size: 512 KiB
HC WP Group Size: 8 MiB
User Capacity: 29.1 GiB
Boot Capacity: 4 MiB ENH
RPMB Capacity: 4 MiB ENH
Boot area 0 is not write protected
Boot area 1 is not write protected

ToradexAI · September 11, 2024, 5:07pm

Hi, I’m the Toradex AI Assistant. The Toradex Support team will follow up with you shortly!

Did you like my answer?

Yes, U-Boot should be able to read an SD card on the Verdin iMX8M Plus. Based on the U-Boot output you provided, it seems that U-Boot is not detecting the SD card as MMC Device 0. Here are a few steps you can take to troubleshoot this issue:

Check Device Tree Configuration: Ensure that the device tree configuration for the SD card interface is correct. The SD card interface should be properly defined in the device tree source (DTS) files. Verify that the relevant nodes are enabled and correctly configured.
Check U-Boot Configuration: Make sure that U-Boot is configured to support the SD card interface. This includes enabling the appropriate drivers and ensuring that the SD card interface is not disabled in the U-Boot configuration.
Check Hardware Connections: Since you mentioned that the SD card works in Linux and Tezi, it is less likely to be a hardware issue. However, double-check the hardware connections and ensure that the SD card interface has the correct pull-up resistors and other necessary components.
Use mmc rescan Command: You mentioned that you tried the mmc rescan command, but it did not help. Ensure that you are using the correct device number for the SD card. For example, if the SD card is on FSL_SDHC: 1, you should use:
```
mmc dev 1
mmc rescan
mmc info
```
Check U-Boot Logs: Look for any error messages or warnings in the U-Boot logs that might indicate why the SD card is not being detected. This can provide clues about what might be going wrong.
Update U-Boot: Ensure that you are using the latest version of U-Boot provided by Toradex. There might be bug fixes or improvements in newer versions that could resolve the issue.

If you have verified all the above steps and the issue persists, it might be helpful to reach out to Toradex support with detailed information about your custom carrier board design and the steps you have taken so far. They might be able to provide more specific guidance based on your hardware and configuration.

Please, also check if these links can help you solve your problem:

External storage 32GB on a carrier board with Colibri iMX7, iMX8.

alex.tx · September 11, 2024, 7:28pm

Definitely you need to do:

mmc dev 1
mmc rescan
mmc info

Another thing I noticed from the boot log you provided is the build date (Jan 01 1970):
U-Boot 2022.04-6.4.0+git.dc27426aa417 (Jan 01 1970 - 00:00:00 +0000)

It seems like you’re using a custom version of U-Boot. If the commands above do not help, please try using the Toradex-built U-Boot.

matthijs · September 12, 2024, 9:01am

Ah, found part of it. Apparently the default mmc selected device is mmc dev 0, which does not exist. Explicitly selecting mmc dev 1 helps, at least on a Dahlia board:

Verdin iMX8MP # mmc dev 1
switch to partitions #0, OK
mmc1 is current device
Verdin iMX8MP # mmc rescan
Verdin iMX8MP # mmc info
Device: FSL_SDHC
Manufacturer ID: 3
OEM: 5744
Name: WP32G 
Bus Speed: 200000000
Mode: UHS SDR104 (208MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 29.7 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes

Seeing this, I also recall that the verdin datasheet says that there are three USDHC interfaces, and the first one (mmc dev 0 I presume) is connected to the wifi, second to SD card and third to eMMC. So that actually makes sense.

So that part was PEBCAK all along :‐)

It still breaks on our custom carrierboard, but at least I can now get u-boot to scan the SD card, which shows activity on the SD card pins, so now I have something to scope and debug. I’ll report back with what I found (hopefully because I solved it).

For completeness, here’s a bit of reply about the u-boot versions that I already typed and might still be useful for future context.

I am pretty sure I am not using a self-compiled u-boot. The log above was taken from a Verdin module that has been floating around our office for a while, so it might still have a u-boot from an older Toradex-supplied image, but I see the same behavior on a brand-new image that I flashed a recent Toradex clean image onto (and then updated the OS with torizon-core-builder, but AFAIU that does not change u-boot). That newer module has:

U-Boot SPL 2022.04-6.5.0+git.28dc906f6107 (Jan 01 1970 - 00:00:00 +0000)
U-Boot 2022.04-6.5.0+git.28dc906f6107 (Jan 01 1970 - 00:00:00 +0000)

To doublecheck, I just booted Tezi and had it install Torizon OS Easy Pairing version 6.7.0+build.18.modified.container (2024-07-05)

That gives me:

U-Boot SPL 2022.04-6.7.0+git.7588eb559ca2 (Jan 01 1970 - 00:00:00 +0000)
U-Boot 2022.04-6.7.0+git.7588eb559ca2 (Jan 01 1970 - 00:00:00 +0000)

alex.tx · September 12, 2024, 4:14pm

It’s very strange when I flashed 6.7.0 build I’ve got :

U-Boot SPL 2022.04-6.7.0+git.7588eb559ca2 (May 28 2024 - 11:19:14 +0000)
U-Boot 2022.04-6.7.0+git.7588eb559ca2 (May 28 2024 - 11:19:14 +0000)

Did you install the image from our server or from external storage (USB/SD)?

matthijs · September 13, 2024, 9:56am

From the server. I booted tezi using uuu then selected “Torizon OS Easy Pairing version 6.7.0+build.18.modified.container (2024-07-05)” from the list (which I presume is taken from the server).

I might be running an older version of Tezi though, I just used whatever I had already downloaded before (Verdin-iMX8MP_ToradexEasyInstaller_5.7.3+build.17). I can try using a more recent Tezi next week, maybe that makes a difference.

But anyway, this just a detail that, AFAICS, is not really relevant to the issue (since the same module and u-boot version works ok on the dahlia board).

I did a bit more debugging yesterday, and it seems that on both boards, u-boot goes through pretty much the same motions, but things look a bit different. I am not familiar enough with the SD card protocol to see what the problem is. I’ll share some scope images below, maybe someone else recognizes something in there?

In the following scope images, the yellow channel is the CLK pin, blue is CMD. I did not scope CS, I should probably do that for a next test. I added a decoder for the data sent, but I think I configured it for falling edge instead of rising edge sampling, so do not look to closely at that.

Here’s what the first activity looks on the Dahlia board, which works:

I think this starts with a ton of clock toggles to start up the SD card, then waits a ms or so, and then sends one command. This is then (not shown on this scope image) followed by repeated pairs of commands (which I think might be polling for the SD card to be ready with something, since after about 170 ms (150 tries or so) the response to the second command changes, a third command is given and then (after some delay) the actual data transfer seems to start. Since these first two transactions (as shown above) are different on our custom carrierboard already, I’ll focus on those (but I have more images of the later transactions too).

The rampup seems to just toggle the clock with CMD high, but with increasing voltage (maybe to gauge the voltagek where the card starts responding?), I do not have a zoomed scope image of that.

Zooming in on the second transaction (which I think is the first command given):

I think this might be an all-zeroes command (CMD0?) followed by one byte of checksum (should be 48bits) followed by one or two bytes of response from the card (which is all ones, so maybe the card is not responding). I suspect this might be a command to switch to SPI mode, but AFAIU that involves pulling CS low during the command, which I did not check.

What I think is weird about this is that:

The specs say start bits are 01, but the first bits I see here are 101. What’s up with that leading 1? It seems to be present in all transactions I’ve scoped, though.
AFAIU a command is 48 bits and a response is 48 or 136 bits. So why is this transaction only 50-60 bits in total?

Anyway, on our custom carrierboard, things look different:

This also shows a rampup, but the shape is different (it actually looks more sensible to me, being a linear rampup). This suggests electrical differences, but maybe there are scoping artifacts (I did not have good scoping points for these signals).

Looking at the second transaction, it also ramps up a little, unlike the same transaction on the dahlia. Ignore the protocol decoding, I think it is synced incorrectly as well as having wrong polarity. The total number of bits is about 61 (that’s what I counted, I might have miscounted one or two), so this actually looks like the same transaction as on the Dahlia, but with different voltages.

The comparing the next transaction (showing only the first of two commands) this is the Dahlia:

while this is our carrier board:

These now look more sensible in terms of length (about 48 bits command and 136 bits response). It looks like the command sent is the same, but on the dahlia the SD card responds, but on our board it does not (all ones).

Here’s the relevant parts of our custom carrierboard design (which is essentially identical to the dahlia and reference schematic, except maybe the card detect pin):

Does this ring a bell with anyone? Can someone tell me if those ramp-ups in voltage are intended and how they should look?

I’m planning to do a bit more scoping next week, probably look at voltage rails to see if there is anything weird there (though all supplies are on way before this happens) and see what the CS pin does.

alex.tx · September 13, 2024, 6:18pm

I don’t want to delve into the SD card protocol details. We have no issues with SD card communication on any of our carrier boards, so we can assume everything is functioning correctly in that regard.

Instead, I would rather attribute the issue to your carrier board implementation. It could be a signal integrity issue due to the board layout (please refer to our Carrier Board Layout Guide for the relevant requirements).

Or more likely, it’s an issue with the SD card’s power supply.

Initially, the Verdin SD-related pins (like any other Verdin pins) are configured for 1.8V output. When the SDIO interface is enabled at the software level, the Verdin internal power rails for the SD interface switch from 1.8V to 3.3V, which explains the ramp-up you’re observing. This also applies to the SD_1_PWR_EN signal.

The AP22653A switch used in your design has a significantly higher Enable Threshold Voltage compared to the MIC94073YMT-TR switch that we use in our designs. This difference may cause a delay in the 3.3V_uSD rail voltage ramp-up, which could result in the SD card not accepting the initial part of communication due to insufficient power at that moment.

matthijs · September 17, 2024, 4:10pm

Thanks, that was exactly the bit I needed to move this forward.

You were almost right, our switch is indeed slower and that is likely the problem, but it’s not the threshold that’s different (both have 1.5V worst-case IIUC), but the turn on delay time that is significantly longer for our switch (max 4ms). Here’s how it looks when I manually turn on the switch on our board (pink is switch enable, blue is switch output):

This has a 2ms delay plus a 200us ramp, which is pretty much expected (datasheet says 4ms max delay, 300us typical 1ms max ramp).

Comparing to the dahlia (this is on SD-card insertion with linux booted, since manually controlling the switch was not easy):

This has a 750us delay and 300us ramp, which is a bit faster than the datasheet suggests (for the MIC94073 used), which could be because the voltage and the load current ID are lower.

When scoping the same thing when doing mmc dev 1 in u-boot, it looks like this for our carrierboard (also showing SD clock in yellow):

And like this on Dahlia:

I suspect that the voltage ramp that the SOC puts on the clock or other pins leaks through the SD card back to the 3V3 rail. And I suspect that both load switches have slightly different current sinking behavior on their output when they are still turned off, which could explain the difference in rampup shape.

In any case, I am quite confident that the longer turn-on time of our switch is indeed the problem. The definitive way to verify this would be to shorten the turn-on time (but I am not sure how), or to increase the u-boot delay (but that requires recompiling u-boot, which I can manage but would need to dive into).

However, I wonder if increasing the u-boot delay might actually be a good idea for Toradex to include in their official u-boot builds? AFAICS the current delay between the first bunch of clock toggles and the first actual command is just fairly arbitrary, I believe that is this 1ms delay: mmc.c « mmc « drivers - u-boot-toradex.git - U-Boot bootloader for Apalis and Colibri modules combined with the time it takes to do the first clock toggles, u-boot effectively waits 2ms from enable pin high to the first actual command. The switch used in the Toradex designs seems to manage a stable 3V3 after 1.1ms, so that works in practice. But according to the datasheet, the worst-case delay+rampup is 2.5ms, which could be too long in some circumstances.

I think it would be good to consider adding an extra delay for this, maybe even 5ms to also allow for slower switches like ours? It seems u-boot even has support for this via the regulator-ramp-delay device-tree property, which I think is already present here but set to 100us and could be enlarged (in hindsight looking at previous scope images, there is indeed a 100us delay between regulator enable toggle and the first clock toggles).

Would you be open to changing this value to, say, 5000us and include that in future torizon OS builds? I am happy to test this change (maybe even submit a PR, but I probably should pivot my time spending away from this issue now and come back to it later).

matthijs · September 30, 2024, 1:35pm

@alex.tx (or someone else), did you see my above request for a change in Toradex’ u-boot config? If you are open to this change, we can likely move forward with our current hardware, if not we might need to look into hardware changes for this.

alex.tx · September 30, 2024, 3:51pm

Please post your request here: Feature Requests - Toradex Community

matthijs · September 30, 2024, 6:29pm

Thanks, done: u-boot: Increase regulator-ramp-delay for SD-card regulator