State / State Transition

Description

Event Severity

Problem

Action if Any

→ TODWAIT (Initializing)

Time of day wait

INFO

Case 1: Not an issue

Case 2: Proceed to Action

Case 1: AP is just turned on or time source is just switched → wait for up to 3 minutes for TOD_Wait to transition to ACQUIRING.

Case 2: Waiting for 3 minutes or longer.

Ensure that the configured PTP leader of the AP is online and reachable.

If you have a secondary PTP server configured for the AP, after 5 minutes the AP will fallback to the secondary PTP and attempt to retrieve a time sync

If above fails:

- Check whether the PTP Server is reachable on the network.

- Check whether the PTP Server is successfully GPS synced.

- Check if the PTP Server can be reached by Celona Edge Server (Execute Ping command to GM from the new “Diagnostic Tools” feature on Celona Orchestrator)

TODWAIT → PTPACQUIRING

After time of day is recovered, AP is actively establishing synchronization with its PTP leader as a reference. We expect that an AP should remain in this state for approximately 20 minutes.

CRITICAL

Proceed to Action, if Acquiring > 20min

If the AP is taking more than 20 minutes to transition to the FREQ_LOCK / PHASE_LOCK state, then please check for the following:

From a layer 2 perspective, we strongly recommend the best performance to ensure the APs are connected on a one-switch hop from the PTP server.

If the network conditions mentioned above are good, and the state is still ACQUIRING please contact Celona Support.

PTPACQUIRING → PTPFREQLOCK
​
​

This state indicates that the frequency of the AP’s clock has been adjusted and synchronized to match the frequency of the PTP leader. Frequency lock ensures that the local clock isn't running faster or slower than the reference over time, but it doesn't necessarily mean that they are in phase. Achieving frequency lock is often the first step in the synchronization process. It ensures that the clocks tick at the same rate.

WARN

Proceed to Action, if state persists > 2min

If the state FREQ_LOCK persists for over a 2-minute period, there might be an issue. If there is >30ms jitter in the network, then this could impact achieving PHASE LOCK (phase synchronization). Celona recommend the APs be connected on a one switch hop from the PTP server.

If network conditions mentioned above are good, and the state is still FREQ_LOCK please contact Celona Support.

PTPFREQLOCK → PTPPHASELOCK

PTPACQUIRING ->

PTPPHASELOCK

After achieving frequency lock, the next step is to synchronize the phase. In the PHASE_LOCK state, both the frequency and the phase of AP are synchronized to the PTP leader. This means that the local clock not only ticks at the same rate as the reference but also ticks at the same time.

INFO

NO

No action

PTPPHASELOCK → PTPFREQLOCK

This means that the local clock no longer ticks at the same time as the PTP leader.

WARN

Proceed to Action

If the AP is returning to FREQ_LOCK from PHASE_LOCK refer to the Action for the state “FREQ_LOCK -> PHASE_LOCK”.

If there is >30ms jitter in the network, then this could impact achieving PHASE LOCK. Jitter can be measured by doing a ping test on the

Celona recommends the APs be connected on a one-switch hop from the PTP server.

If the network conditions mentioned above are good, and the state is still FREQ_LOCK please contact Celona Support.

PTPPHASELOCK → PTPHOLDOVER

If the AP goes into a Holdover state it has lost connection to the PTP Leader. Without this reference, the clock can't maintain accurate synchronization over extended periods The AP will maintain its clock with its oscillator which means that over time the AP’s clock will drift away from the PTP Leader’s clock which may impact the network’s performance.

CRITICAL

Proceed to Action

If the PTP External GM or PTP Leader AP has unstable GPS time sync or the PTP time source is unreachable after being synced or previously reachable, the follower APs synced to it may go into a Holdover state. Without this reference, the clock can't maintain accurate synchronization over extended periods The AP will maintain its clock with its own oscillator which means that over time the AP’s clock will drift away from the PTP Leader’s clock which may impact the network’s performance.

Case 1: Secondary PTP Source Configured

In this state, the APs will continue to be RF Operational, and devices continue to stay connected. The holdover source can ensure 8 microseconds of holdover from 8 to 24 hours, but due to LTE TDD networks having stringent timing requirements, + 1.5 microseconds, we attempt to recover the network much quicker.

If the above scenario, the APs will be automatically moved to a Secondary PTP GM configured for redundancy.

After 30 minutes of being unable to recover from the Holdover state. The first AP is moved to the secondary server after 30 minutes of being in Holdover state, with every other AP being moved at 5-minute intervals after to reduce impact to connected devices.

Note that deployments are planned for RF redundancy to ensure a device sees at least 2 APs it can successfully connect to.

Case 2: No Secondary PTP source

If a Secondary PTP source for redundancy is not configured on the Celona Orchestrator, and if,

the PTP External GM or PTP Leader AP has unstable GPS time sync or the PTP time source is unreachable after being synced or previously reachable, the follower APs synced to it may go into a Holdover state. In this state, the APs will continue to be RF Operational, and devices will continue to stay connected. The holdover source can ensure 8 microseconds of holdover from 8 to 24 hours and the system continues to retry obtaining PTP sync again in the meantime. After 24 hours, if the PTP sync state still hasn't recovered from Holdover the customer may start observing some devices failing mobility between APs or a reduction in Uplink throughput due to the clock drifting.

Celona recommends the APs be connected on a one-switch hop from the PTP server.

If the network conditions mentioned are good, and the state is still HOLDOVER please contact Celona Support.

Case 1: PTP Time Source is a Trimble GM or similar.

- Check whether GM is reachable on the network.

- Check whether GM is successfully GPS synced.

- Check if GM can be reached by Celona Edge Server (Execute Ping command to GM from new “Diagnostic Tools” feature on Celona Orchestrator)

Case 2: PTP Time Source is a GPS-synced Access Point on the same site.

- Check if the Access Point is reachable on the network.

- Check whether the Access Point is successfully GPS synced.

PTPHOLDOVER -> PTPACQUIRING

PTPHOLDOVER -> PTPPHASELOCK

The AP has regained connection to a PTP Leader and is attempting to regain synchronization has already regained it

If Acquiring,

CRITICAL

Proceed to Action, if Acquiring > 20min

If the state is PHASE LOCK, no action. If the state is stuck on Acquiring for >20min then please check for the following:

From a layer 2 perspective, we strongly recommend the best performance to ensure the APs are connected on a one-switch hop from the PTP server.

If the network conditions mentioned above are good, and the state is still ACQUIRING please contact Celona Support.

REFFAILED

REFFAILEDSOAK

No sync reports have been received from the PTP time source soak time has passed and no active PTP server can be found anymore. This can be observed if the Access Point is in Holdover for too long.

No sync reports have been received from the PTP time source and AP is still in soak time; REF_FAILED_SOAK will be followed by REF_FAILED state.

This can be observed if the Access Point is in Holdover for too long.

CRITICAL

Proceed to Action

Case 1: Secondary PTP Source Configured

In this scenario, the APs will be automatically moved to a Secondary PTP GM configured for redundancy.

After 30 minutes of being unable to recover from the Holdover state. The first AP is moved to the secondary server after 30 minutes of being in Holdover state, with every other AP being moved at 5-minute intervals after to reduce impact to connected devices.

Note that deployments are planned for RF redundancy to ensure a device sees at least 2 APs it can successfully connect to.

Case 2: No Secondary PTP source

If a Secondary PTP source for redundancy is not configured on the Celona Orchestrator, and if,

the PTP External GM or PTP Leader AP has unstable GPS time sync or the PTP time source is unreachable after being synced or previously reachable, the follower APs synced to it may go into Holdover state. In this state, the APs will continue to be RF Operational, and devices will

continue to stay connected. The holdover source can ensure 8 microseconds of a holdover from 8

to 24 hours and the system continues to retry obtaining PTP sync again in the meantime. After 24 hours, if the PTP sync state still hasn't recovered from Holdover the customer may start observing some devices failing mobility between APs or a reduction in Uplink throughput due to the clock drifting.

This AP will continue to stay in Holdover and will eventually go to a REF_FAILED state where no more sync reports can be received.

Contact Celona Support if no Secondary PTP server is configured and the REF_FAILED state is observed.

State / State Transition

Description

Event Severity

Problem

Action if Any

GPSSYNCCONFIGURED

Access Point has been configured to use GPS as a Time source

INFO

N/A

N/A

GPSSYNCSUCCESSFUL ->

GPSSYNCSRCNOTAVAILABLE

GPSSYNCCONFIGURED ->

GPSSYNCSRCNOTAVAILABLE

GPS sync source is not available or unstable and Access Point configured to use GPS as a Time source or previously successfully GPS synced can sync to it.

CRITICAL

Proceed to Action

If the GPS synchronization is lost on an AP that was previously successfully time synced, the Access Point continues to stay in a state of Holdover on previously known GPS-obtained timing. In this state, the APs will continue to be RF Operational, and devices will stay connected. The holdover source can ensure 8 microseconds of holdover from 8 to 24 hours and the system continues to retry obtaining GPS sync again in the meantime. After 24 hours, if the GPS sync state still hasn't recovered the customer may start observing some devices failing mobility between APs or reduction in Uplink throughput due to the clock drifting.

- Check whether the GPS antenna on the Access Point is soundly connected.

- Check if the GPS cable is not bent.

- If the above is okay, contact Celona Support.

GPSSYNCSUCCESSFUL

INFO

N/A

N/A