In 2005, the APS Safety Interlocks (SI) Group began an internal review of their first-generation personnel safety systems (PSS-G1). The review was to cover every aspect of the safety system. The results of the review necessitated an upgrade path for the 44 beamlines utilizing the generation 1 PSS’s. The scope of the upgrade addressed PLC and software obsolescence, conformity to DOE orders and other industrial and international standards, standardizing the way the programmable controller (PLC) code was written, replacing unreliable components causing frequent system trips, creating intuitive EPICS screens to speed up the troubleshooting process, creating a complete set of new drawings for each beamline, developing an acceptance test system, designing a new, non-invasive validation system, and reducing validation times from two to three days to one day or less.
In July, 2007, the SI Group was given permission to use X-ray Science Division beamline 6-BM-A,B, a then-non-operational beamline, as a test bed to develop the prototype upgraded PSS. In August 2007, the project proposal was approved by APS management, and in October 2007, the development of a new PLC acceptance test simulator began. This would give the programmers the ability to test their code before it was downloaded at the beamline, thus reducing the risk of finding errors halfway through a validation and having to start over. It also provided more in-depth testing of off-normal actions and actually verified the functional truth tables created in the specifications.
In December 2007, the PSS-G1 upgrade was presented to the Beamline Safety Design Review Steering Committee (BSDRSC), Radiation Safety Policy and Procedures Committee (RSPPC) and other members of the laboratory safety hierarchy. In Feb 2008, the SI Group was given approval to proceed with the “development and test” installation at 6-BM. In July 2008, the first code was downloaded and functional testing began.
To address an increase of failures of process controllers monitoring radiation safety system cooling water flows that were causing increased beamline down time, a mini upgrade of the PSS was implemented prior to the full upgrade and took two years to complete. Modifications to the Chain B PLC code were also implemented at this time to address some shortcomings.
Safety Interlocks Group PSS safety software programmers were tasked with modifying the unique code on the chain A and chain B processors, and producing procedures that would verify the code before it went out to the field for validation. The software modifications were significant and required a concentrated and sustained programming effort in order to complete the project on schedule. After the software modifications were complete, each programmer acceptance-tested their code on a test stand in the lab to beamline specific procedures. The programmers completed the necessary software development, implementation, testing, and deployment for validation on schedule.
Another engineer was responsible for creating every validation procedure. That engineer reviewed the beamline-specific user requirements document and create a procedure that checks every input, every output, and multiple beamline functional tests that would verify the upgrade was installed and implemented properly. Another engineer was tasked with scheduling and overseeing the fabrication, assembly, and installation tasks performed by the technicians between and during shutdown. He was also responsible for ordering ten's of thousands of dollars of parts and components and getting them on site on time to allow the techs enough time to build and QA the equipment required for installation during the shutdowns.
Safety Interlocks Group technicians were tasked with fabricating, assembling, and installing the components required for these upgrades. Each technician was required to take Institute of Printed Circuits (IPC) soldering and wire termination standards training and pass tests, then become proficient in producing high-quality connections and terminations. This resulted in very-high-quality installed panels and Human-Machine Interfaces (HMI's), lessening checkout times and creating installations that will be trouble free for years. The technicians also provided valuable feedback and lessons learned after each install. This feedback was then applied to make the future installs that much more efficient.
The full upgrade began in Dec 2009 and only two to three could be installed during each of the three shutdowns that occurred each year. All 44 beamlines were completed in May 2016. Over 350 panels were fabricated, assembled, QA’d and installed. Over 275 shutters were re-wired, many replaced with radiation hardened cable. Over 1500 drawings and 250 documents were created or modified, verified and archived in ICMS. A total of 88 versions of PLC code were developed, reviewed, acceptance tested and validated. Every item from the review was addressed and implemented, and validation times reduced to a day or less.
By Greg Markovich (AES, gmm@anl.gov)