d29a8134c7
The PTP hardware semaphore (PFTSYN_SEM) is used to synchronize
operations that program the PTP timers. The operations involve issuing
commands to the sideband queue. The E810 does not have a hardware
sideband queue, so the admin queue is used. The admin queue is slow.
I have observed delays in hundreds of milliseconds waiting for
ice_sq_done.
When phc2sys reads the time from the ice PTP clock and PFTSYN_SEM is
held by a task performing one of the slow operations, ice_ptp_lock can
easily time out. phc2sys gets -EBUSY and the kernel prints:
ice 0000:XX:YY.0: PTP failed to get time
These messages appear once every few seconds, causing log spam.
The E810 datasheet recommends an algorithm for reading the upper 64 bits
of the GLTSYN_TIME register. It matches what's implemented in
ice_ptp_read_src_clk_reg. It is robust against wrap-around, but not
necessarily against the concurrent setting of the register (with
GLTSYN_CMD_{INIT,ADJ}_TIME commands). Perhaps that's why
ice_ptp_gettimex64 also takes PFTSYN_SEM.
The race with time setters can be prevented without relying on the PTP
hardware semaphore. Using the "ice_adapter" from the previous patch,
we can have a common spinlock for the PFs that share the clock hardware.
It will protect the reading and writing to the GLTSYN_TIME register.
The writing is performed indirectly, by the hardware, as a result of
the driver writing GLTSYN_CMD_SYNC in ice_ptp_exec_tmr_cmd. I wasn't
sure if the ice_flush there is enough to make sure GLTSYN_TIME has been
updated, but it works well in my testing.
My test code can be seen here:
https://gitlab.com/mschmidt2/linux/-/commits/ice-ptp-host-side-lock-10
It consists of:
- kernel threads reading the time in a busy loop and looking at the
deltas between consecutive values, reporting new maxima.
- a shell script that sets the time repeatedly;
- a bpftrace probe to produce a histogram of the measured deltas.
Without the spinlock ptp_gltsyn_time_lock, it is easy to see tearing.
Deltas in the [2G, 4G) range appear in the histograms.
With the spinlock added, there is no tearing and the biggest delta I saw
was in the range [1M, 2M), that is under 2 ms.
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Reviewed-by: Przemek Kitszel <przemyslaw.kitszel@intel.com>
Signed-off-by: Michal Schmidt <mschmidt@redhat.com>
Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
28 lines
771 B
C
28 lines
771 B
C
/* SPDX-License-Identifier: GPL-2.0-only */
|
|
/* SPDX-FileCopyrightText: Copyright Red Hat */
|
|
|
|
#ifndef _ICE_ADAPTER_H_
|
|
#define _ICE_ADAPTER_H_
|
|
|
|
#include <linux/spinlock_types.h>
|
|
#include <linux/refcount_types.h>
|
|
|
|
struct pci_dev;
|
|
|
|
/**
|
|
* struct ice_adapter - PCI adapter resources shared across PFs
|
|
* @ptp_gltsyn_time_lock: Spinlock protecting access to the GLTSYN_TIME
|
|
* register of the PTP clock.
|
|
* @refcount: Reference count. struct ice_pf objects hold the references.
|
|
*/
|
|
struct ice_adapter {
|
|
/* For access to the GLTSYN_TIME register */
|
|
spinlock_t ptp_gltsyn_time_lock;
|
|
|
|
refcount_t refcount;
|
|
};
|
|
|
|
struct ice_adapter *ice_adapter_get(const struct pci_dev *pdev);
|
|
void ice_adapter_put(const struct pci_dev *pdev);
|
|
|
|
#endif /* _ICE_ADAPTER_H */
|