Downtime Manager

(Clear)
Choose Parameters
Time
Taxonomy
Incident ID

Downtime Events

{1 - 10 of 6,588}
Type Title Down Duration
Reviewed
SME | OPR

HLB Hall B: Torus Magnet issue 18-Mar-2024 17:30 1 hour 28 minutes  

Event ID: 18894

Timeline

Hall B: Torus Magnet issue
(Down)
18-Mar-2024 17:30
18-Mar-2024 17:47
18-Mar-2024 18:05
18-Mar-2024 18:22
18-Mar-2024 18:40
(Up)
18-Mar-2024 18:58
Restore: 0 seconds

Incidents

Description Period Cause Log Entries
ID:* 12900 Review:* SME: No OPR: No Title: Hall B: Torus Magnet issue Summary: Ramping up of Torus magnet issue Duration: 1 hour 28 minutes Review Level: Level Ⅱ Time Down: 18-Mar-2024 17:30 Time Up: 18-Mar-2024 18:58 Category: Hall Downtime System: Hall Downtime Component: Controlled Access

HLC Hall C: Refurbishment Plan 17-Mar-2024 16:14 26 hours 44 minutes  

Event ID: 18891

Timeline

Hall C: Refurbishment Plan
(Down)
17-Mar-2024 16:14
17-Mar-2024 21:34
18-Mar-2024 02:55
18-Mar-2024 08:16
18-Mar-2024 13:37
(Up)
18-Mar-2024 18:58
Restore: 0 seconds

Incidents

Description Period Cause Log Entries
ID:* 12894 Review:* SME: No OPR: No Title: Hall C: Refurbishment Plan Summary: Refurbishment Plan elog4267802 Duration: 26 hours 44 minutes Review Level: Level Ⅲ+ Time Down: 17-Mar-2024 16:14 Time Up: 18-Mar-2024 18:58 Category: Hall Downtime System: Hall Downtime Component: Controlled Access

ACC Vacuum issues downstream side of Lambertson + Pass 3 Separator Floating Deck Unit Swap + 2L22-2 FCC Swap (4) 16-Mar-2024 10:52 53 hours 50 minutes  

Event ID: 18874

Timeline

Vacuum spikes no coming down in a few 4T0(n) regions and 2C
1L21 Dropped out of HV
Pass 3 Separator Floating Deck Unit Swap
2L22-2 FCC Swap
(Down)
16-Mar-2024 10:52
16-Mar-2024 21:38
17-Mar-2024 08:24
17-Mar-2024 19:10
18-Mar-2024 05:56
(Up)
18-Mar-2024 16:42
Restore: 0 seconds

Incidents

Description Period Cause Log Entries
ID:* 12884 Review:* SME: No OPR: No Title: Vacuum spikes no coming down in a few 4T0(n) regions and 2C Summary: Vacuum not coming down in a few of the regions 4T0{8,9,...} an 2C. Later...4Tn recovered but 2C didn't. Looks like it is downstream of lambertson Duration: 49 hours 59 minutes Review Level: Level Ⅲ+ Time Down: 16-Mar-2024 10:52 Time Up: 18-Mar-2024 12:51 Category: Software System: IOC (Low Level) Applications Component: Vacuum
ID:* 12885 Review:* SME: No OPR: No Title: 1L21 Dropped out of HV Summary: After turning HV back on, 1L21 recovered briefly, then dropped out of HV. Tried turning back on, and HV did turn on, but then dropped out, with no explanation. Tried bypassing cavity 8 to see if that would help, but no dice. Called RF on call, and they are looking at it now. Duration: 2 hours 54 minutes Review Level: Level Ⅱ Time Down: 16-Mar-2024 16:40 Time Up: 16-Mar-2024 19:34 Category: Cold RF System: Cryomodules Component: 1L20
ID:* 12896 Review:* SME: No OPR: No Title: Pass 3 Separator Floating Deck Unit Swap Summary: Pass 3 sep tripped off at start of owls on an LCW fault. RF's investigation this morning found a bad FDU that needed to be swapped, work still underway. Duration: 9 hours 42 minutes Review Level: Level Ⅲ+ Time Down: 18-Mar-2024 07:00 Time Up: 18-Mar-2024 16:42 Category: RF System: RF Separators Component: RRF6E00A
ID:* 12897 Review:* SME: No OPR: No Title: 2L22-2 FCC Swap Summary: Cavity not making CRFP when recovering SL RF. RF On-call says FCC needs to be swapped, work is underway. Duration: 1 hour 7 minutes Review Level: Level Ⅱ Time Down: 18-Mar-2024 12:57 Time Up: 18-Mar-2024 14:04 Category: Cold RF System: RF Cavities Component: 2L22-2

ACC MBC7S02H 15-Mar-2024 21:00 14 minutes  

Event ID: 18873

Timeline

MBC7S02H Repeated Mismatches
(Down)
15-Mar-2024 21:00
15-Mar-2024 21:02
15-Mar-2024 21:05
15-Mar-2024 21:08
15-Mar-2024 21:11
(Up)
15-Mar-2024 21:14
Restore: 0 seconds

Incidents

Description Period Cause Log Entries
ID:* 12883 Review:* SME: No OPR: Yes Title: MBC7S02H Repeated Mismatches Summary: MBC7S02H is unable to hold setpoint. Will mismatch to low with no faults presenting, take demand setpoint and cycle without issue, only to repeat this sequence a few moments later. OPSMCC swapped trim card. Duration: 14 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 21:00 Time Up: 15-Mar-2024 21:14 Category: Correctors System: Correctors (iron core) Component: MBC7S02H

ACC Gangphase and Pathlength 15-Mar-2024 19:06 54 minutes  

Event ID: 18893

Timeline

Gangphase and Pathlength
(Down)
15-Mar-2024 19:06
15-Mar-2024 19:16
15-Mar-2024 19:27
15-Mar-2024 19:38
15-Mar-2024 19:49
(Up)
15-Mar-2024 20:00
Restore: 7 minutes

Incidents

Description Period Cause Log Entries
ID:* 12899 Review:* SME: No OPR: Yes Title: Gangphase and Pathlength Summary: Only xD receiving nA-beam, but getting frequent ion chamber trips. Placing dumplettes in to touch-up gangphase and pathlength and allow main machine orbit and energy locks to converge before turning them off and resuming CW delivery to xD. Duration: 47 minutes Review Level: Level Ⅱ Time Down: 15-Mar-2024 19:06 Time Up: 15-Mar-2024 19:53 Category: Beam Transport System: Beam Transport Component: Beam Quality Check

ACC 2L20-3 Instability (2) 15-Mar-2024 17:20 54 minutes  

Event ID: 18871

Timeline

2L20-3 Instability
Locks Left On During nA Beam Delivery
(Down)
15-Mar-2024 17:20
15-Mar-2024 17:30
15-Mar-2024 17:41
15-Mar-2024 17:52
15-Mar-2024 18:03
(Up)
15-Mar-2024 18:14
Restore: 22 minutes

Incidents

Description Period Cause Log Entries
ID:* 12881 Review:* SME: No OPR: Yes Title: 2L20-3 Instability Summary: 2L20-3 constantly tuning, not converging. Attempted an open-loop tune, and during ended up adjusted RATN. After this, started experiencing forward power spikes, so GSET-zero. Duration: 15 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 17:20 Time Up: 15-Mar-2024 17:35 Category: Cold RF System: RF Cavities Component: 2L20-3
ID:* 12898 Review:* SME: No OPR: Yes Title: Locks Left On During nA Beam Delivery Summary: Mis-steering main machine orbits. Switched to tune-mode to dumplettes to hone orbits back in. Duration: 17 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 17:55 Time Up: 15-Mar-2024 18:12 Category: Operations System: Operations Personnel Component: Operator Error

HLC HMS Q2 Interlock 15-Mar-2024 17:14 46 hours 59 minutes  

Event ID: 18872

Timeline

HMS Q2 Interlock
(Down)
15-Mar-2024 17:14
16-Mar-2024 02:37
16-Mar-2024 12:01
16-Mar-2024 21:25
17-Mar-2024 06:49
(Up)
17-Mar-2024 16:13
Restore: 3 hours 13 minutes

Incidents

Description Period Cause Log Entries
ID:* 12882 Review:* SME: No OPR: Yes Title: HMS Q2 Interlock Summary: Interlock failure prevents sending beam. Duration: 43 hours 46 minutes Review Level: Level Ⅲ+ Time Down: 15-Mar-2024 17:14 Time Up: 17-Mar-2024 13:00 Category: Hall Downtime System: Hall Downtime Component: Hardware Failure

ACC TAG5C PS trip on DC Overvoltage 15-Mar-2024 14:38 43 minutes  

Event ID: 18865

Timeline

TAG5C PS trip on DC Overvoltage
(Down)
15-Mar-2024 14:38
15-Mar-2024 14:46
15-Mar-2024 14:55
15-Mar-2024 15:03
15-Mar-2024 15:12
(Up)
15-Mar-2024 15:21
Restore: 6 minutes

Incidents

Description Period Cause Log Entries
ID:* 12879 Review:* SME: No OPR: Yes Title: TAG5C PS trip on DC Overvoltage Summary: TAG5C BoxPS tripped on a DC overvoltage fault. Given time of day, AESDC contacted to have a look. AESDC cycled the power of TAG5C and were able to bring BoxPS up and cycle it through hysterisis without issue. Possibly, an operating transistor blew a fuse, which if an isolated component will not affect the output current of the supply. Duration: 37 minutes Review Level: Level Ⅱ Time Down: 15-Mar-2024 14:38 Time Up: 15-Mar-2024 15:15 Category: Box Supply System: Box Power Supplies Component: TAG5C

ACC SL RF short upset (trucks, weather, etc?) 15-Mar-2024 11:29 11 minutes  

Event ID: 18863

Timeline

SL RF Disturbance, Seems to be Vibration
(Down)
15-Mar-2024 11:29
15-Mar-2024 11:31
15-Mar-2024 11:33
15-Mar-2024 11:35
15-Mar-2024 11:37
(Up)
15-Mar-2024 11:40
Restore: 0 seconds

Incidents

Description Period Cause Log Entries
ID:* 12877 Review:* SME: Yes OPR: Yes Title: SL RF Disturbance, Seems to be Vibration Summary: At roughly the same time, several SL C30/50 cavities tripped off: 2L06-{5,6}, 2L09-6, 2L10-{1,2,7}, 2L13-{2,4,7,8}, 2L15-{4,7}. CPS HVs for these zones were stable. 40MVA voltages stable as well. A handful of SL C100s also dropped out of SELAP, but did not completely trip off: 2L22-3, 2L24-5, 2L25-{4,5}. Looking at SL C100 CFQEs, no out-of-the-ordinary tuning had to occur around the time like would be seen if there were a large truck vibrating the cavities or tree cutting in the surrounding area. All cavities, with the exception of 2L10-7, were able to be recovered without intervention. 2L10-7 had to be open-loop tuned. What happened here? Duration: 11 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 11:29 Time Up: 15-Mar-2024 11:40 Category: Nature System: Weather / Environment Component: Construction in Surrounding Area

ACC South Linac Ion Pumps Tripped Off 14-Mar-2024 18:13 3 hours 3 minutes  

Event ID: 18862

Timeline

Beamline Vacuum Ion Pumps 2L14A through 2L17B Tripped Off
(Down)
14-Mar-2024 18:13
14-Mar-2024 18:49
14-Mar-2024 19:26
14-Mar-2024 20:02
14-Mar-2024 20:39
(Up)
14-Mar-2024 21:16
Restore: 26 minutes

Incidents

Description Period Cause Log Entries
ID:* 12876 Review:* SME: No OPR: Yes Title: Beamline Vacuum Ion Pumps 2L14A through 2L17B Tripped Off Summary: Beamline vacuum ion pumps from 2L14A through 2L17B tripped off, closing surrounding valves. iocsl4 remotely rebooted as part of troubleshooting, but wouldn't come back up. Local reboot also did not recover. OPSACG assisted, recovering IOC. After, ion pumps still tripped, so AESIC investigated and found a faulty PS for the ion pumps, a multi-output 6GeV Vacuum PS. Supply's +15V rail producing only 6.9V. PS swapped with a spare and ion pumps powered back on, valve interlocks satisfied, beam recovery begun. AESIC work performed under ATLis 111294. Duration: 2 hours 37 minutes Review Level: Level Ⅱ Time Down: 14-Mar-2024 18:13 Time Up: 14-Mar-2024 20:50 Category: Beamline Vacuum System: Beamline Vacuum Instrumentation Component: Ion Pumps (VIP) Instrumentation
Event Information
  • Append Incident Title
Incident Description
  • Same as Event
Incident Period
Incident Cause
Filter by Category/System
Subject Matter Expert (SME) Review
  • (space separated)
Operability (OPR) Review
Note: An Elog will be generated for Create/Close incident actions
( required)

  • (Level Ⅰ)
    No Yes Reassign

  • (Level Ⅱ)

  • (Level Ⅲ+)

Subject Matter Expert (SME)
  • Example user 1
    Example user 2
Operability (OPR)