Node Time Not Synchronized With Cluster¶

Problem Description¶

The time stamp of a node is not synchronized with the other nodes in this cluster.

Relevant Alerts and Errors¶

Alert

NTP synchronization issues will raise the NodeClockNotSynchronising alert. This alert will be displayed on the Alerts tab of the admin UI. If configured to, this will also be received by email.

Error

The following error means there is a time difference between the server and the client. This is an error that is seen on the client side when trying to interact with ARTESCA via the S3 API.

Error: RequestTimeTooSkewed:
   The difference between the request time and the server's time is too large.
   status code: 403

Root Causes¶

When a node in a 3-node cluster encounters issues such as freezing or crashing, the node’s time and date may no longer be synchronized with the other nodes in that cluster.
When there is a reliability issue with the configured NTP sources. For example, if only two sources are configured, chronyd can stop synchronizing if it considers the two sources as not agreeing with each other, even if they are perfectly reachable.

Investigation Steps¶

Follow these steps to investigate the issue.

Ensure that the chronyd service is active and running.
[artesca-os@bootstrap ~]$
systemctl is-active chronyd
```
active
```
The output must show active. If it is not active, you can enable it with the following command:
[artesca-os@bootstrap ~]$
sudo systemctl enable chronyd

Check service status.

[artesca-os@bootstrap ~]$

systemctl status chronyd

● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-04-02 07:48:13 UTC; 42min ago

The output must show Active: active (running). If the service is not running, you can check logs from last start attempt.

[artesca-os@bootstrap ~]$

sudo journalctl _SYSTEMD_INVOCATION_ID=`systemctl show -p InvocationID --value chronyd.service`

-- Logs begin at Wed 2024-09-25 13:06:08 UTC, end at Tue 2024-10-01 11:14:43 UTC. --
Oct 01 11:14:35 artesca-node chronyd[314810]: chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTE>
Oct 01 11:14:35 artesca-node chronyd[314810]: Fatal error : Could not parse server directive at line 2 in file /etc/chrony>
Oct 01 11:14:35 artesca-node chronyd[314785]: Could not parse server directive at line 2 in file /etc/chrony.conf

If the logs from the last start attempt do not show any relevant information, get all logs.

[artesca-os@bootstrap ~]$

sudo journalctl -xeu chronyd

Oct 01 11:14:35 artesca-node systemd[1]: Stopped NTP client/server.
-- Subject: Unit chronyd.service has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit chronyd.service has finished shutting down.
Oct 01 11:14:35 artesca-node systemd[1]: Starting NTP client/server...
-- Subject: Unit chronyd.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit chronyd.service has begun starting up.
Oct 01 11:14:35 artesca-node chronyd[314810]: chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTE>
Oct 01 11:14:35 artesca-node chronyd[314810]: Fatal error : Could not parse server directive at line 2 in file /etc/chrony>
Oct 01 11:14:35 artesca-node chronyd[314785]: Could not parse server directive at line 2 in file /etc/chrony.conf
Oct 01 11:14:35 artesca-node systemd[1]: chronyd.service: Control process exited, code=exited status=1
Oct 01 11:14:36 artesca-node systemd[1]: chronyd.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit chronyd.service has entered the 'failed' state with result 'exit-code'.
Oct 01 11:14:36 artesca-node systemd[1]: Failed to start NTP client/server.
-- Subject: Unit chronyd.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit chronyd.service has failed.
--
-- The result is failed.

Ensure that all sources are online.

[artesca-os@bootstrap ~]$

chronyc activity

OK
sources online
sources offline
sources doing burst (return to online)
sources doing burst (return to offline)
sources with unknown address

Ensure that chrony is connected to at least one source. At least one source must be shown as synchronized (* in the S column).

Note

If you have just restarted the service, it can take a few minutes to synchronize with a source.

[artesca-os@bootstrap ~]$

chronyc sources -v

.-- Source mode  '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- ip212-227-232-161.pbiaas>     2   6    17     7   -512us[+7158us] +/-   60ms
^- 80.67.184.129                 2   6    17     8  -1566us[+6111us] +/-   22ms
^* time.cloudflare.com           3   6    17     8    +19us[+7701us] +/- 9668us
^- silas.dioptre.fr              2   6    17     7    -17ms[-9020us] +/-   34ms

Ensure that chrony shows a normal status for the system clock.

[artesca-os@bootstrap ~]$

chronyc tracking

Reference ID    : C23ACA14 (sth1.ntp.netnod.se)
Stratum         : 2
Ref time (UTC)  : Tue Apr 02 08:32:27 2024
System time     : 0.000003753 seconds fast of NTP time
Last offset     : +0.000001838 seconds
RMS offset      : 0.000027322 seconds
Frequency       : 4.732 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.005 ppm
Root delay      : 0.002780697 seconds
Root dispersion : 0.000441081 seconds
Update interval : 256.1 seconds
Leap status     : Normal

Leap status must not have a value of Normal.

Problem Resolution¶

After your investigation, take the relevant step to resolve the issue:

Change the time source.

Restart the chronyd service.

[artesca-os@bootstrap ~]$

sudo systemctl restart chronyd