Downtime Events
{1 - 10 of 6,588}
Type
Title
Down
Duration
Reviewed
SME | OPR
SME | OPR
HLB Hall B: Torus Magnet issue 18-Mar-2024 17:30 1 hour 28 minutes
Closed by: Humphry, Chris (chumphry)
Event ID: 18894
Timeline
Hall B: Torus Magnet issue
(Down)
18-Mar-2024 17:30
18-Mar-2024 17:30
18-Mar-2024 17:47
18-Mar-2024 18:05
18-Mar-2024 18:22
18-Mar-2024 18:40
(Up)
18-Mar-2024 18:58
18-Mar-2024 18:58
Restore: 0 seconds
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12900 Review:* SME: No OPR: No | Title: Hall B: Torus Magnet issue Summary: Ramping up of Torus magnet issue | Duration: 1 hour 28 minutes Review Level: Level Ⅱ Time Down: 18-Mar-2024 17:30 Time Up: 18-Mar-2024 18:58 | Category: Hall Downtime System: Hall Downtime Component: Controlled Access |
HLC Hall C: Refurbishment Plan 17-Mar-2024 16:14 26 hours 44 minutes
Closed by: Humphry, Chris (chumphry)
Event ID: 18891
Timeline
Hall C: Refurbishment Plan
(Down)
17-Mar-2024 16:14
17-Mar-2024 16:14
17-Mar-2024 21:34
18-Mar-2024 02:55
18-Mar-2024 08:16
18-Mar-2024 13:37
(Up)
18-Mar-2024 18:58
18-Mar-2024 18:58
Restore: 0 seconds
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12894 Review:* SME: No OPR: No | Title: Hall C: Refurbishment Plan Summary: Refurbishment Plan elog4267802 | Duration: 26 hours 44 minutes Review Level: Level Ⅲ+ Time Down: 17-Mar-2024 16:14 Time Up: 18-Mar-2024 18:58 | Category: Hall Downtime System: Hall Downtime Component: Controlled Access |
ACC Vacuum issues downstream side of Lambertson + Pass 3 Separator Floating Deck Unit Swap + 2L22-2 FCC Swap (4) 16-Mar-2024 10:52 53 hours 50 minutes
Closed by: Humphry, Chris (chumphry)
Event ID: 18874
Timeline
Vacuum spikes no coming down in a few 4T0(n) regions and 2C
1L21 Dropped out of HV
Pass 3 Separator Floating Deck Unit Swap
2L22-2 FCC Swap
(Down)
16-Mar-2024 10:52
16-Mar-2024 10:52
16-Mar-2024 21:38
17-Mar-2024 08:24
17-Mar-2024 19:10
18-Mar-2024 05:56
(Up)
18-Mar-2024 16:42
18-Mar-2024 16:42
Restore: 0 seconds
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12884 Review:* SME: No OPR: No | Title: Vacuum spikes no coming down in a few 4T0(n) regions and 2C Summary: Vacuum not coming down in a few of the regions 4T0{8,9,...} an 2C. Later...4Tn recovered but 2C didn't. Looks like it is downstream of lambertson | Duration: 49 hours 59 minutes Review Level: Level Ⅲ+ Time Down: 16-Mar-2024 10:52 Time Up: 18-Mar-2024 12:51 | Category: Software System: IOC (Low Level) Applications Component: Vacuum | |
ID:* 12885 Review:* SME: No OPR: No | Title: 1L21 Dropped out of HV Summary: After turning HV back on, 1L21 recovered briefly, then dropped out of HV. Tried turning back on, and HV did turn on, but then dropped out, with no explanation. Tried bypassing cavity 8 to see if that would help, but no dice. Called RF on call, and they are looking at it now. | Duration: 2 hours 54 minutes Review Level: Level Ⅱ Time Down: 16-Mar-2024 16:40 Time Up: 16-Mar-2024 19:34 | Category: Cold RF System: Cryomodules Component: 1L20 | |
ID:* 12896 Review:* SME: No OPR: No | Title: Pass 3 Separator Floating Deck Unit Swap Summary: Pass 3 sep tripped off at start of owls on an LCW fault. RF's investigation this morning found a bad FDU that needed to be swapped, work still underway. | Duration: 9 hours 42 minutes Review Level: Level Ⅲ+ Time Down: 18-Mar-2024 07:00 Time Up: 18-Mar-2024 16:42 | Category: RF System: RF Separators Component: RRF6E00A | |
ID:* 12897 Review:* SME: No OPR: No | Title: 2L22-2 FCC Swap Summary: Cavity not making CRFP when recovering SL RF. RF On-call says FCC needs to be swapped, work is underway. | Duration: 1 hour 7 minutes Review Level: Level Ⅱ Time Down: 18-Mar-2024 12:57 Time Up: 18-Mar-2024 14:04 | Category: Cold RF System: RF Cavities Component: 2L22-2 |
ACC MBC7S02H 15-Mar-2024 21:00 14 minutes
Closed by: Merz, Michael (mmerz)
Event ID: 18873
Timeline
MBC7S02H Repeated Mismatches
(Down)
15-Mar-2024 21:00
15-Mar-2024 21:00
15-Mar-2024 21:02
15-Mar-2024 21:05
15-Mar-2024 21:08
15-Mar-2024 21:11
(Up)
15-Mar-2024 21:14
15-Mar-2024 21:14
Restore: 0 seconds
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12883 Review:* SME: No OPR: Yes | Title: MBC7S02H Repeated Mismatches Summary: MBC7S02H is unable to hold setpoint. Will mismatch to low with no faults presenting, take demand setpoint and cycle without issue, only to repeat this sequence a few moments later. OPSMCC swapped trim card. | Duration: 14 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 21:00 Time Up: 15-Mar-2024 21:14 | Category: Correctors System: Correctors (iron core) Component: MBC7S02H |
ACC Gangphase and Pathlength 15-Mar-2024 19:06 54 minutes
Closed by: Moser, Daniel (moser)
Event ID: 18893
Timeline
Gangphase and Pathlength
(Down)
15-Mar-2024 19:06
15-Mar-2024 19:06
15-Mar-2024 19:16
15-Mar-2024 19:27
15-Mar-2024 19:38
15-Mar-2024 19:49
(Up)
15-Mar-2024 20:00
15-Mar-2024 20:00
Restore: 7 minutes
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12899 Review:* SME: No OPR: Yes | Title: Gangphase and Pathlength Summary: Only xD receiving nA-beam, but getting frequent ion chamber trips. Placing dumplettes in to touch-up gangphase and pathlength and allow main machine orbit and energy locks to converge before turning them off and resuming CW delivery to xD. | Duration: 47 minutes Review Level: Level Ⅱ Time Down: 15-Mar-2024 19:06 Time Up: 15-Mar-2024 19:53 | Category: Beam Transport System: Beam Transport Component: Beam Quality Check |
ACC 2L20-3 Instability (2) 15-Mar-2024 17:20 54 minutes
Closed by: Shenenberger, Alex (alexs)
Event ID: 18871
Timeline
2L20-3 Instability
Locks Left On During nA Beam Delivery
(Down)
15-Mar-2024 17:20
15-Mar-2024 17:20
15-Mar-2024 17:30
15-Mar-2024 17:41
15-Mar-2024 17:52
15-Mar-2024 18:03
(Up)
15-Mar-2024 18:14
15-Mar-2024 18:14
Restore: 22 minutes
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12881 Review:* SME: No OPR: Yes | Title: 2L20-3 Instability Summary: 2L20-3 constantly tuning, not converging. Attempted an open-loop tune, and during ended up adjusted RATN. After this, started experiencing forward power spikes, so GSET-zero. | Duration: 15 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 17:20 Time Up: 15-Mar-2024 17:35 | Category: Cold RF System: RF Cavities Component: 2L20-3 | |
ID:* 12898 Review:* SME: No OPR: Yes | Title: Locks Left On During nA Beam Delivery Summary: Mis-steering main machine orbits. Switched to tune-mode to dumplettes to hone orbits back in. | Duration: 17 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 17:55 Time Up: 15-Mar-2024 18:12 | Category: Operations System: Operations Personnel Component: Operator Error |
HLC HMS Q2 Interlock 15-Mar-2024 17:14 46 hours 59 minutes
Closed by: Humphry, Chris (chumphry)
Event ID: 18872
Timeline
HMS Q2 Interlock
(Down)
15-Mar-2024 17:14
15-Mar-2024 17:14
16-Mar-2024 02:37
16-Mar-2024 12:01
16-Mar-2024 21:25
17-Mar-2024 06:49
(Up)
17-Mar-2024 16:13
17-Mar-2024 16:13
Restore: 3 hours
13 minutes
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12882 Review:* SME: No OPR: Yes | Title: HMS Q2 Interlock Summary: Interlock failure prevents sending beam. | Duration: 43 hours 46 minutes Review Level: Level Ⅲ+ Time Down: 15-Mar-2024 17:14 Time Up: 17-Mar-2024 13:00 | Category: Hall Downtime System: Hall Downtime Component: Hardware Failure |
ACC TAG5C PS trip on DC Overvoltage 15-Mar-2024 14:38 43 minutes
Closed by: Thomas, Dillon (dthomas)
Event ID: 18865
Timeline
TAG5C PS trip on DC Overvoltage
(Down)
15-Mar-2024 14:38
15-Mar-2024 14:38
15-Mar-2024 14:46
15-Mar-2024 14:55
15-Mar-2024 15:03
15-Mar-2024 15:12
(Up)
15-Mar-2024 15:21
15-Mar-2024 15:21
Restore: 6 minutes
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12879 Review:* SME: No OPR: Yes | Title: TAG5C PS trip on DC Overvoltage Summary: TAG5C BoxPS tripped on a DC overvoltage fault. Given time of day, AESDC contacted to have a look. AESDC cycled the power of TAG5C and were able to bring BoxPS up and cycle it through hysterisis without issue. Possibly, an operating transistor blew a fuse, which if an isolated component will not affect the output current of the supply. | Duration: 37 minutes Review Level: Level Ⅱ Time Down: 15-Mar-2024 14:38 Time Up: 15-Mar-2024 15:15 | Category: Box Supply System: Box Power Supplies Component: TAG5C |
ACC SL RF short upset (trucks, weather, etc?) 15-Mar-2024 11:29 11 minutes
Closed by: Thomas, Dillon (dthomas)
Event ID: 18863
Timeline
SL RF Disturbance, Seems to be Vibration
(Down)
15-Mar-2024 11:29
15-Mar-2024 11:29
15-Mar-2024 11:31
15-Mar-2024 11:33
15-Mar-2024 11:35
15-Mar-2024 11:37
(Up)
15-Mar-2024 11:40
15-Mar-2024 11:40
Restore: 0 seconds
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12877 Review:* SME: Yes OPR: Yes | Title: SL RF Disturbance, Seems to be Vibration Summary: At roughly the same time, several SL C30/50 cavities tripped off: 2L06-{5,6}, 2L09-6, 2L10-{1,2,7}, 2L13-{2,4,7,8}, 2L15-{4,7}. CPS HVs for these zones were stable. 40MVA voltages stable as well. A handful of SL C100s also dropped out of SELAP, but did not completely trip off: 2L22-3, 2L24-5, 2L25-{4,5}. Looking at SL C100 CFQEs, no out-of-the-ordinary tuning had to occur around the time like would be seen if there were a large truck vibrating the cavities or tree cutting in the surrounding area. All cavities, with the exception of 2L10-7, were able to be recovered without intervention. 2L10-7 had to be open-loop tuned. What happened here? | Duration: 11 minutes Review Level: Level Ⅰ Time Down: 15-Mar-2024 11:29 Time Up: 15-Mar-2024 11:40 | Category: Nature System: Weather / Environment Component: Construction in Surrounding Area |
ACC South Linac Ion Pumps Tripped Off 14-Mar-2024 18:13 3 hours 3 minutes
Closed by: Forman, Eric (eforman)
Event ID: 18862
Timeline
Beamline Vacuum Ion Pumps 2L14A through 2L17B Tripped Off
(Down)
14-Mar-2024 18:13
14-Mar-2024 18:13
14-Mar-2024 18:49
14-Mar-2024 19:26
14-Mar-2024 20:02
14-Mar-2024 20:39
(Up)
14-Mar-2024 21:16
14-Mar-2024 21:16
Restore: 26 minutes
Incidents
Description | Period | Cause | Log Entries | |
---|---|---|---|---|
ID:* 12876 Review:* SME: No OPR: Yes | Title: Beamline Vacuum Ion Pumps 2L14A through 2L17B Tripped Off Summary: Beamline vacuum ion pumps from 2L14A through 2L17B tripped off, closing surrounding valves. iocsl4 remotely rebooted as part of troubleshooting, but wouldn't come back up. Local reboot also did not recover. OPSACG assisted, recovering IOC. After, ion pumps still tripped, so AESIC investigated and found a faulty PS for the ion pumps, a multi-output 6GeV Vacuum PS. Supply's +15V rail producing only 6.9V. PS swapped with a spare and ion pumps powered back on, valve interlocks satisfied, beam recovery begun. AESIC work performed under ATLis 111294. | Duration: 2 hours 37 minutes Review Level: Level Ⅱ Time Down: 14-Mar-2024 18:13 Time Up: 14-Mar-2024 20:50 | Category: Beamline Vacuum System: Beamline Vacuum Instrumentation Component: Ion Pumps (VIP) Instrumentation |
( required)
-
-
-
(Level Ⅰ)No Yes Reassign -
(Level Ⅱ) -
(Level Ⅲ+)