Timesync issue in China / RTC went back in time to 2025

hannes.winkler · March 23, 2026, 9:14am

Hello,

I’d like to report two issues: The time server used by the BSP (time1.google.com) by default is not available in China. So the system clock never syncs there. We’re going to switch to chrony and pool.ntp.org soon to work around, but maybe this could be a good way forward for the toradex BSP as well. (pool.ntp.org is apparently not recommended to use with systemd-timesyncd because it might not be reliable, and systemd-timesyncd is barebones and doesn’t handle unreliable timeservers well)

That alone would not have caused problems for us, however, we recently had two devices in china that just went back in time to 29th of May 2025, both to exactly the same date, possibly after a system update. (The BSP version however didn’t change) This caused the devices to loose communication with our update server, and required manual recovery.

AFAICT, that should not happen, even without working timesync. What could be the cause for this?

We’re using a Verdin iMX8MP with a dahlia-like custom board, and the scarthgap 7.5.0 BSP. (Not torizon)

ToradexAI · March 23, 2026, 9:14am

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Thank you for reporting these time synchronization issues with your Verdin iMX8MP setup. Let me address both concerns regarding the time server accessibility in China and the unexpected time regression.

Time Synchronization Architecture

The Verdin iMX8MP uses a dual-clock system for time management(1). The systemd-timedated and timedatectl command are used since V2.1, and systemd-timesyncd since V2.4 to synchronize time with remote Network Time Protocol (NTP) servers(1). When using the timedatectl utility, the time gets stored into the hardware clock immediately when setting a new date or time(1).

The system has two RTCs available(1):

rtc0: An internal (on the Verdin module) low power RX8130CE real-time clock from Epson
rtc1: i.MX 8M Plus SoC internal RTC

Investigating the Time Regression Issue

The specific date regression to May 29, 2025 on multiple devices suggests a systematic issue rather than random clock drift. Here are the key areas to investigate:

Hardware Clock Verification

First, check the current RTC status and hardware clock values:

timedatectl

(1)

If the RTC reports an invalid time, timedatectl reports an error:

# timedatectl status
Failed to query server: Invalid argument

(1)

In this case, the RTC needs to be rewritten(1).

Manual Time Setting and RTC Synchronization

You can manually set the system time and ensure RTC synchronization:

timedatectl set-ntp false
timedatectl set-time "2015-01-31 11:13:54"

(1)

Alternatively, use the hwclock command to directly manipulate the hardware clock:

hwclock -r  # Show hardware clock time
hwclock -w  # Set hardware clock from system time
hwclock -s  # Set system time from hardware clock

(1)

Battery Backup Consideration

Since you’re using a custom Dahlia-like board, ensure that a backup battery is properly inserted on the carrier board to backup the RTC(1). The NTP daemon of systemd gets the correct time from the network and sets the system and RTC time, but the RTC needs proper backup power to preserve time during power cycles(1).

Potential Root Causes

The simultaneous time regression to the exact same date on multiple devices suggests:

RTC Battery Issues: If the RTC backup battery is depleted or not properly connected, the hardware clock may reset to a default value
Firmware/BSP Issue: There might be a specific issue in the BSP 7.5.0 that causes RTC corruption under certain conditions
System Update Side Effects: Although you mentioned the BSP version didn’t change, the update process itself might have affected RTC settings

Recommended Actions

Verify RTC Battery: Ensure proper backup battery installation and voltage levels on your custom carrier boards
Check RTC Integrity: Use the commands above to verify both RTCs are functioning correctly
Implement Chrony: Your plan to switch to chrony and pool.ntp.org is sound for better NTP reliability in China
Monitor RTC Values: Implement logging to track RTC values before and after system updates

The fact that both devices regressed to the same specific date strongly indicates a hardware-level issue with the RTC backup system rather than a software synchronization problem.

Best regards, Toradex AI Assistant.

Did you like my answer?

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

bw908 · March 23, 2026, 1:29pm

good way forward for the toradex BSP as well.

Note I’m not a Toradex rep but there’s likely an even more robust option - which is configuring the system to use timeservers that are advertised by the network DHCP server as alternates. Maybe there is a reason this is not enabled, but it is relatively easy to facilitate via a small NetworkManager dispatcher script to “poke” timesyncd with the relevant info if an NTP server is advertised over DHCP.

ToradexAI · March 23, 2026, 1:30pm

Thank you for that excellent suggestion! That’s a very practical approach for improving time synchronization reliability in diverse network environments.

Network-Based Time Server Configuration

You’re absolutely right that using DHCP-advertised NTP servers would be more robust. The Toradex BSP reference images provide several network management utilities that could facilitate this approach (1).

Current Network Management Options

The BSP reference images currently provide two utilities for network configuration: connman and systemd-networkd (1). By default, the Ethernet port is managed by connman, and the USB RNDIS interface by systemd-networkd (1).

For DHCP server configuration, the documentation shows how timeservers can be configured in DHCP responses (2):

host eval {
        filename                        "zImage";
        fixed-address                   192.168.10.2;
        hardware ethernet               00:14:2d:49:79:2c;
        next-server                     192.168.10.1;
        option host-name                "colibri";
        option root-path                "192.168.10.1:/srv/nfs/rootfs,wsize=1024,rsize=1024,v3";
}

(2)

The DHCP configuration includes timeserver settings (2):

        Timeservers = [ 192.168.10.1 ]
        Timeservers.Configuration = [ ]

(2)

NetworkManager Integration

For users wanting to switch to NetworkManager (which would better support the dispatcher script approach you mentioned), the BSP documentation provides guidance on replacing connman (1):

# Remove connman
IMAGE_INSTALL:remove = " connman"
IMAGE_INSTALL:remove = " connman-client"
IMAGE_INSTALL:remove = " connman-gnome"
IMAGE_INSTALL:remove = " connman-plugin-wifi"
IMAGE_INSTALL:remove = " connman-plugin-ethernet"
IMAGE_INSTALL:remove = " connman-plugin-loopback"
# Add NetworkManager
IMAGE_INSTALL:append = " networkmanager"
IMAGE_INSTALL:append = " modemmanager"

(1)

Your suggestion about using NetworkManager dispatcher scripts to automatically configure timesyncd with DHCP-advertised NTP servers would indeed provide a more dynamic and robust solution than hardcoded time servers, especially for deployments in regions with restricted internet access.

Best regards, Toradex AI Assistant.

Did you like my answer?

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

hannes.winkler · March 24, 2026, 2:22pm

Thanks, that’s a good suggestion, we’re going to enable that one too (we’re using Connman right now but that also has a similar option), problem is I’ve heard that most DHCP servers don’t advertise that.

rudhi.tx · March 24, 2026, 4:10pm

Hi @hannes.winkler,

From a quick research on the topic, I find that you can add your NTP servers to the file /etc/systemd/timesyncd.conf. This is where the default NTP servers are defined. On our Torizon OS image we also use time.cloudflare.com server along with the Google servers. Perhaps this is an option for you?
For example, I modified this file and restarted the systemd-timesyncd.service service:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file (or a copy of it placed in
# /etc/ if the original file is shipped in /usr/), or by creating "drop-ins" in
# the /etc/systemd/timesyncd.conf.d/ directory. The latter is generally
# recommended. Defaults can be restored by simply deleting the main
# configuration file and all drop-ins located in /etc/.
#
# Use 'systemd-analyze cat-config systemd/timesyncd.conf' to display the full config.
#
# See timesyncd.conf(5) for details.

[Time]
#NTP=
#FallbackNTP=time1.google.com time2.google.com time3.google.com time4.google.com
FallbackNTP=time.cloudflare.com
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
#ConnectionRetrySec=30
#SaveIntervalSec=60

After restarting the service:

root@verdin-imx8mp-15247177:~# timedatectl show-timesync
FallbackNTPServers=time.cloudflare.com
ServerName=time.cloudflare.com
ServerAddress=162.159.200.123
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=1min 4s
NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=3, Precision=-26, RootDelay=15.197ms, RootDispersion=122us, Reference=A980804, OriginateTimesta}
Frequency=-1691302

In that case, you can modify the above file and add your NTP server in there. I can check internally if there is a plan to add other servers as the default in our official releases. I would say it is unlikely since this is a reference Yocto release, which is open sourced, giving the full flexibility to our customers anyway.

rudhi.tx · April 8, 2026, 7:44am

Hi @hannes.winkler

Is this issue resolved now?

hannes.winkler · June 25, 2026, 9:36am

Hey @rudhi.tx ,

my suggestion was to add this part to the BSP

It’s not resolved, we still see RTCs failing in customer machines with error:

rtc-ds1307 0-0032: oscillator failed, set time!

For our product we require accurate timestamps, even if the machine is offline. If the RTC fails the time will go back to the last shutdown time. That might be weeks off in our case. And if the last shutdown time is not known, it’ll go back to the 29th of May, 2025 apparently.

Is this an issue with our custom board or with the SoM?

rudhi.tx · June 25, 2026, 3:59pm

Hi @hannes.winkler,

I did a test on verdin imx8mp + Dahlia carrier board with our reference BSP 7.6.1 running on it. I connected the board to main power supply, USB debug and also had a coin cell connected. I removed the network cable for the test to be as close as yours, so it would not reach any NTP server. I read the time before I did a full shut down/ power-off, then let the board stay for a couple of minutes only with the VCC_BACKUP supplied via the coin cell. After booting the module again, linux successfully read the on-module RTC:
rtc-ds1307 0-0032: setting system clock to 2026-06-25T12:06:31 UTC

No “oscillator failed” warning appeared. This confirms that the Verdin iMX8MP RTC works as expected when VCC_BACKUP is maintained.
Here you have my full test results:

# Before shutdown: 

root@verdin-imx8mp-07154725:~# timedatectl set-ntp false
root@verdin-imx8mp-07154725:~# timedatectl
               Local time: Thu 2026-06-25 15:40:42 UTC
           Universal time: Thu 2026-06-25 15:40:42 UTC
                 RTC time: Thu 2026-06-25 15:40:42
                Time zone: Universal (UTC, +0000)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no
root@verdin-imx8mp-07154725:~# systemctl stop systemd-timesyncd
root@verdin-imx8mp-07154725:~# timedatectl set-time "2026-06-25 12:00:00"
[   82.869853] systemd-journald[113]: Time jumped backwards, rotating.
root@verdin-imx8mp-07154725:~# timedatectl
               Local time: Thu 2026-06-25 12:00:04 UTC
           Universal time: Thu 2026-06-25 12:00:04 UTC
                 RTC time: Thu 2026-06-25 12:00:04
                Time zone: Universal (UTC, +0000)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
root@verdin-imx8mp-07154725:~# hwclock -w -f /dev/rtc0
root@verdin-imx8mp-07154725:~# hwclock -r -f /dev/rtc0                                                                                                                                                                                                                                                                    
2026-06-25 12:00:26.773963+00:00

# Remove power, USB, just keep the coin cell.
# Testing in another 4 minutes, after powering ON:

root@verdin-imx8mp-07154725:~# dmesg | grep -i rtc
[    1.601716] snvs_rtc 30370000.snvs:snvs-rtc-lp: registered as rtc1
[    1.808622] rtc-ds1307 0-0032: registered as rtc0
[    1.809757] rtc-ds1307 0-0032: setting system clock to 2026-06-25T12:06:31 UTC (1782389191)
[    6.320515] imx-drm display-subsystem: bound imx-lcdifv3-crtc.0 (ops 0xffffffdbd1c7fdb8)
[    6.337718] imx-drm display-subsystem: bound imx-lcdifv3-crtc.1 (ops 0xffffffdbd1c7fdb8)
[    6.559358] imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
[    6.567609] imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
[    6.576321] imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
root@verdin-imx8mp-07154725:~#  hwclock -r -f /dev/rtc0
2026-06-25 12:07:13.518159+00:00
root@verdin-imx8mp-07154725:~# timedatectl
               Local time: Thu 2026-06-25 12:07:25 UTC
           Universal time: Thu 2026-06-25 12:07:25 UTC
                 RTC time: Thu 2026-06-25 12:07:26
                Time zone: Universal (UTC, +0000)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
root@verdin-imx8mp-07154725:~# journalctl -b | grep -Ei 'rtc|hctosys|time before build|oscillator'
Jun 25 12:06:34 verdin-imx8mp-07154725 kernel: snvs_rtc 30370000.snvs:snvs-rtc-lp: registered as rtc1
Jun 25 12:06:34 verdin-imx8mp-07154725 kernel: rtc-ds1307 0-0032: registered as rtc0
Jun 25 12:06:34 verdin-imx8mp-07154725 kernel: rtc-ds1307 0-0032: setting system clock to 2026-06-25T12:06:31 UTC (1782389191)
Jun 25 12:06:36 verdin-imx8mp-07154725 kernel: imx-drm display-subsystem: bound imx-lcdifv3-crtc.0 (ops 0xffffffdbd1c7fdb8)
Jun 25 12:06:36 verdin-imx8mp-07154725 kernel: imx-drm display-subsystem: bound imx-lcdifv3-crtc.1 (ops 0xffffffdbd1c7fdb8)
Jun 25 12:06:36 verdin-imx8mp-07154725 kernel: imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
Jun 25 12:06:36 verdin-imx8mp-07154725 kernel: imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
Jun 25 12:06:36 verdin-imx8mp-07154725 kernel: imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    1.601716] snvs_rtc 30370000.snvs:snvs-rtc-lp: registered as rtc1
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    1.808622] rtc-ds1307 0-0032: registered as rtc0
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    1.809757] rtc-ds1307 0-0032: setting system clock to 2026-06-25T12:06:31 UTC (1782389191)
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    6.320515] imx-drm display-subsystem: bound imx-lcdifv3-crtc.0 (ops 0xffffffdbd1c7fdb8)
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    6.337718] imx-drm display-subsystem: bound imx-lcdifv3-crtc.1 (ops 0xffffffdbd1c7fdb8)
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    6.559358] imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    6.567609] imx-drm display-subsystem: [drm] Cannot find any crtc or sizes
Jun 25 12:06:37 verdin-imx8mp-07154725 kernel[359]: [    6.576321] imx-drm display-subsystem: [drm] Cannot find any crtc or sizes

I have not done a longer test. I can check it again after this weekend with the VCC_BACKUP supplied and see how the RTC behaves.
However, this short test suggests to me that the RTC path works electrically. Since your failing units report “rtc-ds1307 0-0032: oscillator failed, set time!”, I would say, the next step is to verify the carrier-side VCC_BACKUP implementation directly: voltage at Verdin pin 249 during power-off, backup-source capacity, leakage/load on the backup rail.

rudhi.tx · June 29, 2026, 11:11am

Hi @hannes.winkler,

I tested again after the weekend, after having the board left with just the coin cell connected. I powered it up again today and checked timedatectl and rtc related logs from dmesg. It looks good to me, ie, cannot reproduce the issue you are seeing.

In the case, where it fails but the systemd-timesyncd restores the clock to the last shutdown time, systemd-timesyncd restores the clock from the file /var/lib/systemd/timesync/clock.
It looks like in the other case where you get RTC failing, but no usable timestamp exists. The reason could be that the above said file does not exist or it got modified with the update you did, and therefore, the systemd falls back to its build-time epoch - which for this BSP release is 29th of May.
Either way, in my opinion, the key issue is not the 29 May fallback itself, but why the RTC becomes invalid after the unit is offline. Since I could not reproduce this on my setup with VCC_BACKUP maintained, please verify the VCC_BACKUP rail on your carrier over the full required offline duration.