Timesync issue in China / RTC went back in time to 2025

Technical Support
, , , ,
1

Hello,

I’d like to report two issues: The time server used by the BSP (time1.google.com) by default is not available in China. So the system clock never syncs there. We’re going to switch to chrony and pool.ntp.org soon to work around, but maybe this could be a good way forward for the toradex BSP as well. (pool.ntp.org is apparently not recommended to use with systemd-timesyncd because it might not be reliable, and systemd-timesyncd is barebones and doesn’t handle unreliable timeservers well)

That alone would not have caused problems for us, however, we recently had two devices in china that just went back in time to 29th of May 2025, both to exactly the same date, possibly after a system update. (The BSP version however didn’t change) This caused the devices to loose communication with our update server, and required manual recovery.

AFAICT, that should not happen, even without working timesync. What could be the cause for this?

We’re using a Verdin iMX8MP with a dahlia-like custom board, and the scarthgap 7.5.0 BSP. (Not torizon)

2

Hi, I’m the Toradex AI Assistant.

I might make mistakes or occasionally get you wrong. Our support team will step in if needed.

Thank you for reporting these time synchronization issues with your Verdin iMX8MP setup. Let me address both concerns regarding the time server accessibility in China and the unexpected time regression.

Time Synchronization Architecture

The Verdin iMX8MP uses a dual-clock system for time management(1). The systemd-timedated and timedatectl command are used since V2.1, and systemd-timesyncd since V2.4 to synchronize time with remote Network Time Protocol (NTP) servers(1). When using the timedatectl utility, the time gets stored into the hardware clock immediately when setting a new date or time(1).

The system has two RTCs available(1):

  • rtc0: An internal (on the Verdin module) low power RX8130CE real-time clock from Epson
  • rtc1: i.MX 8M Plus SoC internal RTC

Investigating the Time Regression Issue

The specific date regression to May 29, 2025 on multiple devices suggests a systematic issue rather than random clock drift. Here are the key areas to investigate:

Hardware Clock Verification

First, check the current RTC status and hardware clock values:

timedatectl

(1)

If the RTC reports an invalid time, timedatectl reports an error:

# timedatectl status
Failed to query server: Invalid argument

(1)

In this case, the RTC needs to be rewritten(1).

Manual Time Setting and RTC Synchronization

You can manually set the system time and ensure RTC synchronization:

timedatectl set-ntp false
timedatectl set-time "2015-01-31 11:13:54"

(1)

Alternatively, use the hwclock command to directly manipulate the hardware clock:

hwclock -r  # Show hardware clock time
hwclock -w  # Set hardware clock from system time
hwclock -s  # Set system time from hardware clock

(1)

Battery Backup Consideration

Since you’re using a custom Dahlia-like board, ensure that a backup battery is properly inserted on the carrier board to backup the RTC(1). The NTP daemon of systemd gets the correct time from the network and sets the system and RTC time, but the RTC needs proper backup power to preserve time during power cycles(1).

Potential Root Causes

The simultaneous time regression to the exact same date on multiple devices suggests:

  1. RTC Battery Issues: If the RTC backup battery is depleted or not properly connected, the hardware clock may reset to a default value
  2. Firmware/BSP Issue: There might be a specific issue in the BSP 7.5.0 that causes RTC corruption under certain conditions
  3. System Update Side Effects: Although you mentioned the BSP version didn’t change, the update process itself might have affected RTC settings

Recommended Actions

  1. Verify RTC Battery: Ensure proper backup battery installation and voltage levels on your custom carrier boards
  2. Check RTC Integrity: Use the commands above to verify both RTCs are functioning correctly
  3. Implement Chrony: Your plan to switch to chrony and pool.ntp.org is sound for better NTP reliability in China
  4. Monitor RTC Values: Implement logging to track RTC values before and after system updates

The fact that both devices regressed to the same specific date strongly indicates a hardware-level issue with the RTC backup system rather than a software synchronization problem.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

3

good way forward for the toradex BSP as well.

Note I’m not a Toradex rep but there’s likely an even more robust option - which is configuring the system to use timeservers that are advertised by the network DHCP server as alternates. Maybe there is a reason this is not enabled, but it is relatively easy to facilitate via a small NetworkManager dispatcher script to “poke” timesyncd with the relevant info if an NTP server is advertised over DHCP.

4

Thank you for that excellent suggestion! That’s a very practical approach for improving time synchronization reliability in diverse network environments.

Network-Based Time Server Configuration

You’re absolutely right that using DHCP-advertised NTP servers would be more robust. The Toradex BSP reference images provide several network management utilities that could facilitate this approach (1).

Current Network Management Options

The BSP reference images currently provide two utilities for network configuration: connman and systemd-networkd (1). By default, the Ethernet port is managed by connman, and the USB RNDIS interface by systemd-networkd (1).

For DHCP server configuration, the documentation shows how timeservers can be configured in DHCP responses (2):

host eval {
        filename                        "zImage";
        fixed-address                   192.168.10.2;
        hardware ethernet               00:14:2d:49:79:2c;
        next-server                     192.168.10.1;
        option host-name                "colibri";
        option root-path                "192.168.10.1:/srv/nfs/rootfs,wsize=1024,rsize=1024,v3";
}

(2)

The DHCP configuration includes timeserver settings (2):

        Timeservers = [ 192.168.10.1 ]
        Timeservers.Configuration = [ ]

(2)

NetworkManager Integration

For users wanting to switch to NetworkManager (which would better support the dispatcher script approach you mentioned), the BSP documentation provides guidance on replacing connman (1):

# Remove connman
IMAGE_INSTALL:remove = " connman"
IMAGE_INSTALL:remove = " connman-client"
IMAGE_INSTALL:remove = " connman-gnome"
IMAGE_INSTALL:remove = " connman-plugin-wifi"
IMAGE_INSTALL:remove = " connman-plugin-ethernet"
IMAGE_INSTALL:remove = " connman-plugin-loopback"
# Add NetworkManager
IMAGE_INSTALL:append = " networkmanager"
IMAGE_INSTALL:append = " modemmanager"

(1)

Your suggestion about using NetworkManager dispatcher scripts to automatically configure timesyncd with DHCP-advertised NTP servers would indeed provide a more dynamic and robust solution than hardcoded time servers, especially for deployments in regions with restricted internet access.

Best regards, Toradex AI Assistant.

Did you like my answer? :+1: :-1:

If I answered your question, please tick the “Solution” checkbox.

For follow-up questions to me, please use @ToradexAI in your post.

5

Thanks, that’s a good suggestion, we’re going to enable that one too (we’re using Connman right now but that also has a similar option), problem is I’ve heard that most DHCP servers don’t advertise that.

6

Hi @hannes.winkler,

From a quick research on the topic, I find that you can add your NTP servers to the file /etc/systemd/timesyncd.conf. This is where the default NTP servers are defined. On our Torizon OS image we also use time.cloudflare.com server along with the Google servers. Perhaps this is an option for you?
For example, I modified this file and restarted the systemd-timesyncd.service service:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file (or a copy of it placed in
# /etc/ if the original file is shipped in /usr/), or by creating "drop-ins" in
# the /etc/systemd/timesyncd.conf.d/ directory. The latter is generally
# recommended. Defaults can be restored by simply deleting the main
# configuration file and all drop-ins located in /etc/.
#
# Use 'systemd-analyze cat-config systemd/timesyncd.conf' to display the full config.
#
# See timesyncd.conf(5) for details.

[Time]
#NTP=
#FallbackNTP=time1.google.com time2.google.com time3.google.com time4.google.com
FallbackNTP=time.cloudflare.com
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048
#ConnectionRetrySec=30
#SaveIntervalSec=60

After restarting the service:

root@verdin-imx8mp-15247177:~# timedatectl show-timesync
FallbackNTPServers=time.cloudflare.com
ServerName=time.cloudflare.com
ServerAddress=162.159.200.123
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=1min 4s
NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=3, Precision=-26, RootDelay=15.197ms, RootDispersion=122us, Reference=A980804, OriginateTimesta}
Frequency=-1691302

In that case, you can modify the above file and add your NTP server in there. I can check internally if there is a plan to add other servers as the default in our official releases. I would say it is unlikely since this is a reference Yocto release, which is open sourced, giving the full flexibility to our customers anyway.

7

Hi @hannes.winkler

Is this issue resolved now?