In 2019/10, noticed the irtfgps1 (Sonoma D12) has a "GPS Reference Time Fault", and at Stratrum 16.
All other status was OK.  Tried rebooting, but the fault remained (and time sync was lost).

subsysreset
Per endrun's suggest did a "subsysreset":

   ssh irtfgps1 -l root
   Sonoma_D12 GPS(root@irtfgps1:~)-> subsysreset

   Bootloader 6010-0070-000 v 1.00 - Dec 27 2012 14:48:55
   FW 6010-0076-000 v  2.20  - Apr 22 2017 13:24:42
   FPGA 6020-0016-000 v 02
   Resetting GPS Receiver...
   Done.
   Sonoma_D12 GPS(root@irtfgps1:~)-> 

This cleared the error. The GPS sync in a few minutes, and returned to Stratum 1.

Firmware Update
Endrun also suggested we update the Firmware (FW) to the latest versions:
   Linux RFS, 6010-0065-000          from v2.40 to v3.04
   GPS Subsystem FW, 6010-0076-000   from v2.20 to v3.04
   GPS Receiver FW, 6010-0081-000    from v1.20 to v1.30

Downloaded the files: 
   Linux RFS, 6010-0065-000        file is: 6010-0065-000.gz
   GPS Subsystem FW, 6010-0076-000 file is: 6010-0076-000.bin
   GPS Receiver FW, 6010-0081-000  file is: 6010-0081-000.bin

Followed "Appendix B - Upgrading the Firmware". For some reason I had trouble with the scp
command and renaming files, but scp to /tmp/ and mv on the gps host worked.

   1. Updated the Linux RFS Version from 2.40 to v3.04. This work:
     on stefan:   
        scp –p ./6010-0065-000.gz root@irtfgps1:/tmp/" on stefan to transfer file to gps unit.
     on irtfgps1: 
        mv /tmp/6010-0065-000.gz /tmp/rootfs.gz 
        upgraderootfs
        updaterootflag 1
        reboot

     "sysversion" now retuns: Sonoma_D12 GPS 6010-0065-000 v 3.04 - Sep 24 22:58:19 2019

   2. Updated the GPS Subsystem FW  from v2.20  to v3.04
     on stefan:   
        scp –p ./6010-0076-000.bin root@irtfgps1:/tmp" on stefan to transfer file to gps unit.
     on irtfgps1: 
        mv /tmp/6010-0076-000.bin /tmp/subsys.bin
        upgradesubsys

     "gpsversion" now retuns: F/W 6010-0076-000 Ver 3.04 - FPGA 6020-0016-000 Ver 03 - SEP 24 16:41:49 2019

   3. Updated the GPS Reciever FW   from v1.20 to v1.30
     on stefan:   
        scp –p 6010-0081-000.bin root@irtfgps1:/tmp/
     on irtfgps1: 
        mv /tmp/6010-0081-000.bin /tmp/rcvr.bin  
        upgradercvr
        subsysreset

        "rcvversion" now say: F/W 6010-0081-000 Ver 1.30 - FPGA 6020-0014-000 Ver 0004 - OCT 2 11:31:55 2019

Reference:
   The email between IRTF and endrun:

   Firmware screen shot before: 
   

   Firmware screen shot after: