How to provide stable time synchronization?

All related to the Kernel, packages system and libraries
Post Reply
wep_hkpu
Posts: 25
Joined: Thu Mar 10, 2016 2:24 pm

How to provide stable time synchronization?

Post by wep_hkpu » Tue Sep 19, 2017 10:07 am

Dear libelium-dev,

I have been trying to solve the problem about incorrect time of our devices for several months.
Please see below forum to see your previous suggestions:
viewtopic.php?f=6&t=23024
viewtopic.php?f=6&t=23606

We have installed 11 Meshlium devices with similar configuration.
The NTP service is configured to be started after booting the devices (in rc.local).
However, the device time can go wrong at some point during its operation and I have no idea about the exact time when it started.
I have to check the device time in the Manager System everyday to see whether the time starts to be incorrect.
I attached a picture to show the monitored time log for your reference.
https://drive.google.com/open?id=0B6r8l ... XNZT3h3ODQ

Suppose that I restarted all the devices on 13/9/17 at 2 a.m.
You can see that the time of more and more devices start to be incorrect.
Once the time is incorrect, the device time will deviate from the actual time in an acceleration rate.
Occasionally, the time can become correct somehow (See device#10 during 13-14/9/17).

It seems abnormal for me as the devices have the same configuration.
Synchronizing the time manually is not sustainable.
I was trying to analyze the cause of the problem but I really have no idea based on the observed results.
Please suggest if you have any ideas about the issue as the time synchronization among all devices is very significant.

Regards.

libelium-dev
Posts: 27967
Joined: Mon Sep 28, 2009 1:06 pm

Re: How to provide stable time synchronization?

Post by libelium-dev » Tue Sep 19, 2017 1:01 pm

Hi wep_hkpu,

Please, execute this command in each machine to check the time synchronization service:

Code: Select all

 ntpq -pn
And post the results.

Regards

wep_hkpu
Posts: 25
Joined: Thu Mar 10, 2016 2:24 pm

Re: How to provide stable time synchronization?

Post by wep_hkpu » Wed Sep 20, 2017 8:08 am

Dear libelium-dev,

I have captured the results from 4 devices.
The first two are from the devices providing incorrect time.

Code: Select all

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 209.58.185.100  130.133.1.10     2 u  773 1024  373    3.963  -565696 24045.1
 118.140.184.98  223.255.185.2    2 u  652 1024  337    4.705  -563576 24047.0
 118.140.184.99  223.255.185.2    2 u  905 1024  275    3.929  -565386 48206.6

Code: Select all

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 118.140.184.99  223.255.185.2    2 u  483 1024  377    3.645  -264998 24088.1
 209.58.185.100  130.133.1.10     2 u  988 1024  377    3.956  -263813 24064.8
The others are from the devices with correct time.

Code: Select all

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*118.140.184.98  223.255.185.2    2 u   34   64  177    4.136  138.436  17.064
+209.58.185.100  130.133.1.10     2 u   50   64  377    4.155  136.572  18.831
+203.95.213.129  195.66.241.10    2 u   53   64  377    5.014  127.257  22.680

Code: Select all

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+203.95.213.129  195.66.241.10    2 u  445  256  376    5.408    5.072   0.155
-37.218.240.198  118.143.17.82    2 u    8  256  377    4.271   -0.955   0.133
+118.140.184.99  223.255.185.2    2 u  194  256  177    4.108    3.069   0.070
*118.140.184.98  223.255.185.2    2 u   56  256  277    4.170    3.195   0.185
Regards.

libelium-dev
Posts: 27967
Joined: Mon Sep 28, 2009 1:06 pm

Re: How to provide stable time synchronization?

Post by libelium-dev » Wed Sep 20, 2017 9:34 am

Hi wep_hkpu,

Let's try to synchronize manually one meshlium that provides wrong time to perform some test, first execute this command to synchonize the clock via internet:

Code: Select all

remountrw && ntpdate pool.ntp.org
Then execute this command to check the clock status:

Code: Select all

ntpq -pn

Code: Select all

date
After that execute this command to synchronize again the date offset:

Code: Select all

 remountrw && hwclock --adjust --debug
Lastly execute again the command:

Code: Select all

ntpq -pn

Code: Select all

date
Please let us your toughts.

Regards

wep_hkpu
Posts: 25
Joined: Thu Mar 10, 2016 2:24 pm

Re: How to provide stable time synchronization?

Post by wep_hkpu » Tue Sep 26, 2017 9:40 am

Dear libelium-dev,

The device time was correct after executing the command.
It can be seen that the time result from 'remountrw && ntpdate pool.ntp.org' is incorrect.
So I have to kill all ntp services first, and run it again.
However, it seems like the device clock has been continued with the wrong pace.

Below is the results. after executing the provided commands.

Code: Select all

SCPxZ4S:~# remountrw && ntpdate pool.ntp.org
26 Sep 07:40:18 ntpdate[27040]: the NTP socket is in use, exiting
SCPxZ4S:~# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 118.140.184.99  118.143.17.82    2 u  534 1024  277    3.941  -207196 24049.3
 203.95.213.129  195.66.241.10    2 u   55 1024  377    4.962  -207308 24072.0
 209.58.185.100  228.35.253.8     2 u   24 1024  377    3.974  -207316 24128.9
SCPxZ4S:~# date
Tue Sep 26 07:40:32 HKT 2017
SCPxZ4S:~# killall ntpd
SCPxZ4S:~# remountrw && ntpdate pool.ntp.org
26 Sep 01:55:08 ntpdate[27095]: step time server 118.140.184.99 offset -20732.531915 sec
SCPxZ4S:~# ntpq -pn
ntpq: read: Connection refused
SCPxZ4S:~# ntpd
SCPxZ4S:~# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 203.95.213.129  195.66.241.10    2 u    2   64    1    3.816  -197.23   0.977
 118.140.184.99  118.143.17.82    2 u    2   64    1    4.301  -189.69   0.977
 213.251.53.11   193.62.22.74     2 u    1   64    1  271.683  -232.53   0.977
SCPxZ4S:~# remountrw && hwclock --adjust --debug
hwclock from util-linux-ng 2.13.1.1
hwclock: Open of /dev/rtc failed, errno=2: No such file or directory.
Using direct I/O instructions to ISA clock.
Last drift adjustment done at 1499616795 seconds after 1969
Last calibration done at 1499616795 seconds after 1969
Hardware clock is on UTC time
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2017/09/25 18:07:01
Hw clock time : 2017/09/25 18:07:01 = 1506362821 seconds since 1969
Time since last adjustment is 6746026 seconds
Need to insert 0 seconds and refer time back 0.000312 seconds ago
Needed adjustment is less than one second, so not setting clock.
SCPxZ4S:~# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 203.95.213.129  195.66.241.10    2 u   16   64    1    3.669  -31.581   0.977
 118.140.184.99  .STEP.          16 u  397   64    0    0.000    0.000   0.977
 213.251.53.11   193.62.22.74     2 u   16   64    1  271.061  -43.521   0.977
SCPxZ4S:~# date
Tue Sep 26 01:55:44 HKT 2017
The time at 01:55:44 was correct.
Then I did wait for about 12 hours and check the time again.
You can notice the time offset below.

Code: Select all

SCPxZ4S:~# ntpdate pool.ntp.org
26 Sep 15:49:21 ntpdate[6211]: the NTP socket is in use, exiting
SCPxZ4S:~# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 203.95.213.129  195.66.241.10    2 u   30   64  377    5.132  -116527 6707.52
 118.140.184.99  223.255.185.2    2 u   63   64  377    3.759  -116449 6761.13
 213.251.53.11   193.62.22.74     2 u   31   64  377  292.916  -116527 6765.59
I don't know how to fix the pace of device's clock.
Please advise if you have some ideas. Thanks.

Regards.

libelium-dev
Posts: 27967
Joined: Mon Sep 28, 2009 1:06 pm

Re: How to provide stable time synchronization?

Post by libelium-dev » Tue Sep 26, 2017 10:26 am

Hi wep_hkpu,

You can program crontab daemon to execute the synchronization command every minute:

Code: Select all

remountrw && crontab -e
Then insert this line in the crontab editor:

Code: Select all

* * * * * remountrw && killall ntpd && ntpdate pool.ntp.org && ntpd && hwclock --adjust --debug
Regards

wep_hkpu
Posts: 25
Joined: Thu Mar 10, 2016 2:24 pm

Re: How to provide stable time synchronization?

Post by wep_hkpu » Tue Sep 26, 2017 12:25 pm

Dear libelium-dev,

Thanks for your suggestion.
I have assigned the command in crontab file but the time isn't correct yet.
Below is the results from log file (the commmand tail -n 50 /var/log/syslog).

Code: Select all

Sep 26 18:35:01 localhost /USR/SBIN/CRON[30834]: (root) CMD (/bin/sh /bin/S3cUR3m0N.sh)
Sep 26 18:35:01 localhost /USR/SBIN/CRON[30839]: (root) CMD (remountrw && killall ntpd && ntpdate pool.ntp.org && ntpd && hwclock --adjust --debug)
Sep 26 18:35:01 localhost sSMTP[30843]: Unable to locate mail
Sep 26 18:35:01 localhost sSMTP[30843]: Cannot open mail:25
Sep 26 18:35:01 localhost /USR/SBIN/CRON[30837]: (root) MAIL (mailed 38 bytes of output but got status 0x0001 )
The command may encounter some errors as the ntp service hasn't been triggered since the last 17 hours.

Code: Select all

SCPxZGS:~# ps aux | grep ntp
root       903  0.0  0.5   4064  1280 ?        Ss   01:53   0:00 ntpd
root     31320  0.0  0.2   3000   708 pts/1    S+   18:39   0:00 grep ntp
Regards.

libelium-dev
Posts: 27967
Joined: Mon Sep 28, 2009 1:06 pm

Re: How to provide stable time synchronization?

Post by libelium-dev » Tue Sep 26, 2017 2:25 pm

Hello,

Could you post the crontab content. Use this command to check it:

Code: Select all

crontab -l
Regards

wep_hkpu
Posts: 25
Joined: Thu Mar 10, 2016 2:24 pm

Re: How to provide stable time synchronization?

Post by wep_hkpu » Mon Oct 09, 2017 6:13 am

Dear libelium-dev,

Below is the content.

Code: Select all

# m h  dom mon dow   command

00 00 * * * /usr/bin/php /mnt/lib/var/www/ManagerSystem/core/API/DB_purge.php >>/mnt/user/logs/db_purge.log 2>&1 &

* * * * * remountrw && killall ntpd && ntpdate pool.ntp.org && ntpd && hwclock --adjust --debug
Regards.

libelium-dev
Posts: 27967
Joined: Mon Sep 28, 2009 1:06 pm

Re: How to provide stable time synchronization?

Post by libelium-dev » Mon Oct 09, 2017 10:58 am

Hi wep_hkpu,

Let us try to use another solution.

First of all, it is necessary to delete the line in the cronjob, using crontab -e and deleting the last line and kill all ntp processes running.

Code: Select all

killall ntpd
Then, you should change the servers in the ntp configuration file (/etc/ntp.conf). It is highly recommended using local servers in order to get the most accurate date and time for your devices. In your case, the server list could be (replace this part of code):

Code: Select all

...
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 3.hk.pool.ntp.org iburst
server 2.hk.pool.ntp.org iburst 
server 3.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
...
Then, you have to force to start the ntp daemon

Code: Select all

/usr/sbin/ntpd -g
and immediatly after, you can see the system logs in order to see how the change is effective

Code: Select all

tail -f /var/log/syslog
Oct  9 00:00:25 localhost ntpd[9545]: ntpd 4.2.4p4@1.1520-o Sun Nov 22 17:34:54 UTC 2009 (1)
Oct  9 00:00:25 localhost ntpd[9546]: precision = 999.000 usec
Oct  9 00:00:25 localhost ntpd[9546]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Oct  9 00:00:25 localhost ntpd[9546]: Listening on interface #1 wildcard, ::#123 Disabled
Oct  9 00:00:25 localhost ntpd[9546]: Listening on interface #2 lo, ::1#123 Enabled
Oct  9 00:00:25 localhost ntpd[9546]: Listening on interface #3 eth0, fe80::20d:b9ff:fe2a:e6c4#123 Enabled
Oct  9 00:00:25 localhost ntpd[9546]: Listening on interface #4 lo, 127.0.0.1#123 Enabled
Oct  9 00:00:25 localhost ntpd[9546]: Listening on interface #5 eth0, 192.168.3.136#123 Enabled
Oct  9 00:00:25 localhost ntpd[9546]: kernel time sync status 0040
Oct  9 00:00:25 localhost ntpd[9546]: frequency initialized -106.731 PPM from /var/lib/ntp/ntp.drift
Oct  9 00:00:34 localhost ntpd[9546]: synchronized to 37.218.240.198, stratum 2
Oct  9 16:24:17 localhost ntpd[9546]: time reset +59022.593300 s
Oct  9 16:24:17 localhost ntpd[9546]: kernel time sync status change 0001
Oct  9 16:24:20 localhost /USR/SBIN/CRON[9589]: (root) CMD (remountrw && killall ntpd && ntpdate pool.ntp.org && ntpd && hwclock --adjust --debug)
Oct  9 16:24:20 localhost sSMTP[9591]: Unable to locate mail
Oct  9 16:24:20 localhost sSMTP[9591]: Cannot open mail:25
Oct  9 16:24:20 localhost /USR/SBIN/CRON[9588]: (root) MAIL (mailed 38 bytes of output but got status 0x0001 )
Oct  9 16:24:43 localhost dnsmasq[3468]: no address range available for DHCP request via eth0
You can see in the lgos how the time is updated.

ntpd service updates automatically time and data from the specified servers. The update frequency is established by the service. It maintains an internal clock quality indicator. If the clock seems stable, updates are less frequent. If the clock seems unstable, updates are more frequent. This algorithm takes some time in working properly because it has to take historical data. if you want to change it, you can add in the configuration file the maximum and minimum value, like this:

Code: Select all

server 3.hk.pool.ntp.org iburst minpoll 4 maxpoll 4
Smallest and largest allowable polling values are 4 (16 seconds) and 17 (1.5 days) respectively. The maximum poll interval defaults is 10 (1,024 s) and the minimum is 6 (64 s).

Let us know if this configuration works in your devices.

Best regards,

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest