CAT Chat Minutes

December 08, 2003


General Information

Machine Stats R. Klaffky:  

Delivered beam:  1326 hrs.
Percent of scheduled time:  97.10
Downtime:  39.61 hrs.
Mean time between faults:  53.06

Machine Fault:  


Machine fault
#64 12 /04 08:00 to 12/05 12:21

28.35 hrs.

0.43 hr. BPLD Trip


Beamline Downtime Summary:   Summary period 11/28 - 12/04/2003











Serious PSS fault on P8 shutter



Fault reset by floor coordinator on-call




Beamline lost ID control



XFE personnel reset from home




Failure of PSS switches on P8 shutter



Engineer and technician were called in; outboard limit switch on MS2 was replaced




Beamline computer malfunction, affecting data collection



User called beamline personnel at home




Orbit misplaced following end of machine studies



Previous orbit was determined and steering completed




BLEPS equipment problem



BLEPS equipment was plugged into non-GFCI outlet

Modifications to Door Lock N. Friedman:   Nick Friedman explained that modifications to door locks are proceeding.

December-January Maintenance Period Update B. Wesolowski:   Below is the updated schedule as presented at CAT Chat.

Work affecting beam-lines during the Dec. Jan. Maintenance Period
(12-23-03 through 1-23-04 )

Switch gear maintenance X4 - Friday Jan. 2, 2004
LOM 434 and Sectors 12, 13, 14, & 15
Power will be off from approximately 6 a.m. to 6 p.m .  
EFOG will contact beam-line personnel and establish 120V power to their area for necessary items.

Switch gear maintenance X5 - Saturday Jan.10, 2004
LOM 435 and Sectors 16, 17, 18, & 19
Power will be off from approximately 6 a.m. to 6 p.m.
EFOG will contact beam-line personnel and establish 120V power to their area for necessary items.

Survey and Alignment:
Map sectors 14, 26 and 27 for girder move in April 2004
Decker Chicane survey sectors 24, 25 & 31

ID Work:  
3-ID:  (3US and 3DS - Remove, re-measure and reinstall ID)
4-ID:  (4DS - Remove, re-measure and reinstall ID)
Calibration and maintenance of all IDs
Install dipole magnets in new ID in sector 21

Relocation of DIW transmitters:  
DIW transmitters relocation at 5-ID, 7-ID & 8-ID
DIW switchover to APS process water:   Sectors 1, 33 & 34

Front End:
1-ID (re-stacking of Pb bricks of the ratchet wall collimator internal SR work)
6-ID (re-stacking of Pb bricks of the ratchet wall collimator internal SR work)
8-BM (install BLEPS to PSS interface, install remote shutter interface)
17-BM (replace x-bpms, remove and replace FEV to all metal seal FEV - need some access in FOE)
18-ID (replace x-bpms, bake out FE system - need some access in FOE)
21-ID (install and test new x- bpms)
32-ID Integrate existing BIV to FE system
33-ID Integrate existing BIV to FE system       

09-ID:  1/5 to 1/7
11-ID:  1/5 to 1/6
14-BM:  1/8 to 1/9
19-BM:  1/12 to 1/13
19-ID:  1/12 to 1/13
22-ID:  1/14 to 1/15
13-BM: 1/19 to 1/20 also support new shutter installation
31-ID:  1/19 to 1/20
20-BM:  1/21 to 1/22

Pending to be scheduled late into SD:   8-BM and 17-BM for validation.
Replace all watchdog timer relays in all PSS systems (except 13BM, 20BM and 31ID recently done).

Modify ACIS wiring for Zones A to E (PSS and FE-EPS Field devices intermittent and possibly causing PSS Faults)
Validate ACIS Zones A to E (Global Offline and causes PSS Faults)
Work on FE components ( citronox water systems):   To Be Determined
Work on ratchet wall collimators:   To Be Determined  
Shutter Work - to be determined

If you require cryo-pump maintenance and/or enclosure door maintenance during this period, please contact your Floor Coordinator to make arrangements.

Attachment to CAT Chat Minutes 12/08/2003:

MCR Ups Report G. Markovich :   As a result of the Controls 1 UPS trips that occurred November 9 th and 10 th Dranetz power monitors were installed on the panelboards feeding all the MCR and computer room UPS's to enable characterizing the incoming power.   Greg's report summarizes the findings and planned actions.

The inverter log on the UPS's indicated there was an event on the system that caused the UPS's to momentarily operate in inverter mode.   The Dranetz power analyzer was installed to pinpoint the source and use the information to mitigate the events.   The event typically occurred around 01:00 Monday and Tuesday mornings, so an HVAC unit was first suspected.   It was also noted that the UPS's performed a self-test every 7 days, always around 01:00 Monday and Tuesday mornings.  







Figures 1, 2 and 3 show a system event occurred Monday morning around 01:00 on A and C phases at an approximately 11 hertz rate that lasted about 4 seconds.   The fact that the event only occurred on A and C phase ruled out a 3 phase load therefore eliminating all conventional loads such as elevators, HVAC or pumps.   The only single phase load on A and C phase is the Controls UPS2.   The inverter log from the Controls UPS2 indicated a self-test was performed at 01:00 Monday morning.   The inverter log from the Controls 1 UPS indicated a line fault occurred at 01:00 .   It appeared the notches on the line are created by the UPS's themselves and cause the other UPS's on the same panelboard to momentarily run in inverter mode.   Suspicions were proven the following day when the events in figures 4, 5 and 6 were captured.







The three preceding figures display characteristics very similar to figures 1, 2 and 3. These figures actually represent the Controls UPS1 performing a self-test and inducing transients on the system. The other UPS's tied to the same panelboard do not always react to the self-test, but the evidence shows they do on a regular basis which in my opinion is setting us up for a failure.

As a result of these findings, the UPS self tests were disabled.  Manually initiated tests will be performed on a regular, controlled basis.  I have contacted the factory to get specific information such as what amplitude and duration line event will cause the UPS to kick into inverter mode and cause a line loss fault and a brownout fault?   Is the amplitude and sensitivity adjustable?   What does the UPS actually do during the self-test that causes the transients on the system?   As of this report, he awaits these answers.

On a somewhat related subject, on November 24 th at 06:39 the Controls UPS1 input breaker tripped causing the UPS to run on batteries for 18 minutes then shut itself down when the batteries ran down. Investigation into this event resulted in identifying a loose breaker-to-buss connection in panelboard TERP-C1 at positions 10 and 12. The breaker was replaced and relocated to positions16 and 18. Figures 7 and 8 below show the damage done by the overheating.   The panelboard interior should be replaced.  Also, as a result of the batteries running down without any notification in the MCR, Bruce Stejskal has installed an EPICS alarm that will indicate when any of the four UPS's go into inverter mode and give the operators an opportunity to respond.