Keeping drinking water flowing and maintaining proper wastewater processing for millions of Americans is no easy task. Municipalities around the country typically have numerous remote pump and water distribution stations that must operate reliably to properly distribute water and manage the treatment of wastewater. To ensure optimal operation, municipalities continue to iterate on the deployment of supervisory control and data acquisition (SCADA) systems that eliminate the need to have operators monitor each station in person around the clock. But, as with any technological advance, remote SCADA HMIs present new challenges for municipalities to manage and host these systems.
While municipalities have experts to manage water distribution and wastewater treatment, they do not always have in-house experts to maintain their SCADA system. As a result, important tasks such as performing upgrades that keep the system running efficiently or focusing on the security of data being transferred are not always a priority. Therefore, working with a third-party integrator that specializes in the implementation and hosting of fully-integrated SCADA systems, such as Lord & Company, can be a more time- and cost-effective approach versus trying to host and maintain your own SCADA system.
Figure 1. An illustration of how all the components of a SCADA system interact.
The Lord & Company Approach: SMS SCADA
Internet of Things (IoT) technology has presented a tremendous opportunity to further simplify SCADA system management and hosting, especially for water and wastewater municipalities. At Lord & Company, we’ve used our decades of experience, talent, and our commonsense approach to SCADA to develop a cost-effective and reliable SCADA HMI hosting solution. The goal of our approach is to make SCADA system HMIs more readily available, interactive, and easier to maintain. Our SMS SCADA solution is based on the following key components:
- Dual-redundant Cloud-based servers hosted by Lord & Company
- Cellular data communication with the secure distributed network protocol (DNP3)
- Familiar, easy-to-use HMI SCADA software
With SMS SCADA, users can easily remotely perform the following functions for any of their pump or water distribution stations:
- Changing the mode, status, runtimes, and rates of all pumps
- Reviewing current and past alarms
- Adjusting setpoints
- Making notes in the operator notebook
- Analyzing trends and data
At Lord & Company, we understand that your time and focus should be on running your plant, not upgrading and maintaining your SCADA system. With SMS SCADA, you can let us take care of the never-ending hardware and software upgrades required to maintain and secure your SCADA system. With our approach based on familiar HMI software, cellular communication, and Cloud-based data storage, you can rest assured that you have a long-lasting, worry-free SCADA solution in place.
New control systems have greater communication requirements than in the past. Tighter regulatory restrictions demand that control data must be passed between areas of the plant. Reduced staffing requires that plant information needs to be accessible anywhere in the plant, or sometimes outside of the plant. Advanced graphical interface tools and data logging place a greater demand on the control system’s communications bandwidth. All this combines to make the communications backbone of a Control System a critical component.
Due to the increased demand for speed and reliability, specially designed industrialized hardware and redundant fiber networks were used. The redundant hardware made sure that any one break in a communication path would not restrict the communications. However, this break needed to be corrected before a second issue occurred. The decision was made to use the computer-based SCADA control system to monitor its own health and alarm if a problem was detected.
Hirschmann’s Rail-Switch family were selected from a variety of vendors providing industrialized, managed Ethernet switches because they supported the Ethernet/IP communication protocols used by the control system and could be integrated directly into the system without the need for additional hardware.
The Trihedral VTScada SCADA application was configured to monitor the health of the network switches. A loss of redundancy would be alarmed on by the computer system in the same place that process alarms (high temperature, low pH, etc.) are handled, providing a visual indication, audible indication, historical log, and remote notification by way of phone, email, or text.
Integrating the switch into the system was a three step process: configuration of the switch, implementing the Hirschmann-provided tools in the Rockwell Automation PLC control system, and configuring the computer-based SCADA software to monitor, display, and log alarm conditions.
Step 1: Configure the Switch
Using Hirschmann’s HiDiscovery software tool, the managed switches were configured for redundant fiber-ring topology. In addition to the basic configuration, each switch was configured to enable the Ethernet/IP protocol and the Signal Contact configured to activate on a loss of Redundancy in the ring.
It is important to note that the HiDiscovery software is provided by Hirschmann with all managed switches. However, the other switch configuration tools like the Web-based interface and the Command Line Interface can also be used.
Step 2: Program the PLC
The EDS (Electronic Device Signature) Files and Rockwell Integration tool were downloaded from the Hirschmann website. The EDS files were added to the Rockwell Software Logix5000 programming software using the EDS Hardware Installation Wizard, allowing the Switch to talk to the PLC processor as if it were an IO module. To establish the communications link between the PLC processor and the switch, the switch was simply added to the Ethernet Network tab as an “IO Module”, specifying only the IP Address of the switch and rate of data exchange.
In order to process the information from the switch, an AOI (Add-on Instruction) developed by Hirschmann was imported into the PLC program. This AOI drives the data transfer between the switch and the control processor, extracting the configuration and status information on the health of the switch directly to the PLC program memory variables.
An instruction was added to the PLC program to check the health of the communications between the PLC and the switch using a GSV instruction to monitor the “Entry Status” and generate a communications alarm.
PLC program logic was created to reference the Hirschmann_Switch AOI in order to manipulate information to/from the switch and populate PLC program variables. Information includes port configurations and status, switch temperature, model & serial numbers, diagnostic statistics, and configuration information. Almost all information available from the switch’s Web or Command Line Interface (CLI) are available.
Step 3: Configure the SCADA software
Finally, tags were created in the VTScada SCADA application to alarm on a loss of communications with the switch, or the loss of redundancy of the fiber ring. When a loss of redundancy occurs, the system will indicate the problem audibly and visually in the alarm summary. Visually, the operator can look at a display showing the entire plant communications and quickly identify the problem and begin the process of fixing the fiber ring.
Utilizing a Hirschmann industrial Ethernet solution with Rockwell Automation control hardware and Trihedral’s SCADA monitoring software, Lord and Company was able to provide a cost-effective solution to increase the reliability of the customer’s Water Treatment plant control system. By tightly integrating control network diagnostics with the control system located at the plant the system health can be monitored in real-time using tools with which the plant personnel are familiar. This ensures that potential problems are brought to the attention of people who can act on the information in a timely manner, preventing interruptions in the treatment process or loss of data needed for regulatory reporting.
Customer: City of Reidsville, NC Public Works
Location: Reid Lake Road, Reidsville, NC
Description: 9 MGD capacity plant pulling from Lake Reidsville
· Rockwell Automation CompactLogix Programmable Controller system
· Hirschmann RS20-0800T1T1SAABHH with Industrial Protocols
· Trihedral VTScada SCADA Software running on industrial computers
Lord & Company is a proven systems integrator providing control, monitoring, and information systems for improved, expanded, or new municipal water and wastewater treatments plants, industrial processes, or radio communication systems.
One critical component of these systems is SCADA technology. To understand the importance of a reliable SCADA solution, we will consider three things: current challenges around SCADA systems, key factors in successful systems, and how to choose a reliable solution partner.
Understand Challenges Surrounding SCADA Systems
In the vast networks of municipal water and wastewater plants, industrial operations, or radio systems, we know that SCADA systems are critical to uphold communication. Effective communication systems help organizations reach to remote locations and relieve those remote locations of the burden of requiring additional manpower. However, the time and costs associated with operating typical SCADA systems cause massive headaches for owners looking for seamless control, monitoring, and information systems.
With these challenges in mind, we created SMS (SCADA Made Simple) to offer an easy-to-use HMI solution that brings the benefits of SCADA technology.
Consider Critical Factors for Successful SCADA Solutions
Because SCADA systems manage many processes and withstand much interaction from operators, organizations need a SCADA solution that is secure, robust, and reliable.
SMS SCADA features a robust user interface that allows for information and data communication between monitored systems and the user. It allows operators to send and receive data, and control alarms and status through a user-friendly user interface. It saves time and money by using a cloud-based platform to mitigate the constant hardware and software upgrades required to maintain typical SCADA systems. With a highly secure data center served by a Tier 1 premium bandwidth supplier, Lord & Company has created an integrated solution with dual redundant servers for a safe, robust, reliable HMI solution
SMS SCADA is supported by our 90,000 sq. ft. data center with up to 25,000 servers. High-security data centers with onsite physical and electronic monitoring feature SSAE-16 and HIPPA compliance, and are Safe Harbor Certified.
Choose a Reliable SCADA Solution Provider
At Lord and Company, we are experienced in RF protocols, wireless technology, and the systems relying on your wireless network. Our teams use combined knowledge to help leading companies effectively use wireless communications for dependable and protected SCADA systems with solutions like SMS SCADA. Our CSIA certification, vast network of industry partnerships, and proven experience with large municipal water and wastewater systems like that of Brunswick County, NC, and fire, police, and EMS radio control systems have lead customers to trust working with us at Lord & Company time and again.
Lord And Company, Inc recently completed 3 Weeks of Motorola Toolbox, STS and Wonderware Intouch training for our highly valued customer in the North East. Training was conducted in co-operation with Motorola Solutions, Inc. 12 technical personnel each week for a total of 36 personnel were trained at a convenient facility in Brooklyn, NY. This training was specific to the customer’s System’s hardware, applications and software. Training format included a daily mixture of instructor Presentation and Hands On Lab (8 Labs) with Instructor support.
- Motorola MOSCAD Basics: The course provided knowledge for Toolbox software application backup methods, basic use of and troubleshooting with the Toolbox software, replacement procedures for hardware and reloading of the application programs and configuration files.
- Motorola MOSCAD Advanced Training: Advanced level provided a detailed understanding of the System and Network Configuration, the data tables and ladder programming with procedures for online monitoring of programs and data tables.
- Introduction to Motorola System Tool Suite ”STS” Software for ACE3600 and the differences between STS and Toolbox.
- Motorola STS/ACE3600 Basics: This course provided knowledge for STS software application backup methods, basic use of and troubleshooting with the STS software, replacement procedures for hardware and reloading of the application programs and configuration files.
- Motorola ACE3600 Advanced Training: Advanced level provided a detailed understanding of the System and Network Configuration, the data tables and ladder programming with procedures for online monitoring of programs and data tables.
- Setup and Troubleshooting with a Protocol Analyzer: The course provided the purpose for and instructions of how to setup a protocol analyzer and how to export and interpret the data.
- SCADA System Hardware, Software and Server Basics: The course provided an understanding of the functions of the hardware and software utilized in the system and how it communicates with the Motorola ACE3600 and MOSCAD processors. Includes procedures for rebooting the systems, making system backups and hard drive and data management.
- Wonderware Intouch Basics: This course provided an understanding of the Tag Dictionary db, event logger, Modbus driver configuration for communicating to the FIU/FEP’s, links to the graphics screens and graphics development.
- Lab Setup:
- Twelve (12) Laptops with Motorola System Tool Suite ”STS” Software for ACE3600 , Motorola Toolbox 9.503 Software and Wonderware Intouch software.
- Six (6) MOSCAD RTU’s with 420 Processor, PS and I/O modules and configured and tested for comm to rental PC’s. One RTU per two students.
- Six (6) ACE3600 RTU’s with Processor, PS and I/O modules and configured and tested for comm to rental PC’s. One RTU per two students.
A Big Thank You to the following Team members:
- Kenny Brooks for configuring the Laptops & PLC’s and setting up the lab in our Fort Mill, SC Training Facility. This setup was invaluable to get the training flow synchronized for optimal learning.
- Brooke Morris, for coordinating the shipment of equipment under complex delivery requirements.
- Our Trainers, James Owens (Week 1), Michael Missaggia (Week 2) and Rafael Balderrama (Week 3) for extraordinary team work, combining all the different materials and co-coordinating amongst each other, so everyone could provide the same level of exceptional training.
A Special mention and Thank You to Dana Albina from Motorola Solutions, Inc for providing Training Site and managing logistics in Brooklyn, NY.
Figure 1: Lab Setup for Internal Testing at Lord And Company, Inc Training Facility.
Credit: Kenny Brooks
Figure 2: Reception Area at Training Facility in Brooklyn, NY.
Credit: Dana Albina, Motorola Solutions, Inc
Figure 3: Training Facility in Brooklyn, NY.
Credit: Dana Albina, Motorola Solutions, Inc
Lab Instruction – Slide Samples
Figure 4: Lab 7: STS Setup – Creating New Project Slide.
Figure 5: Lab 7: STS Setup – Compiling Application Sample.
As always, we tried to meet our Goal of Starting and Finishing with Excellence. With a Great Team and Team Work, I believe WE did.
Project Manager, PMP.
At Lord & Company, we partner with clients to create and sustain industrial automation systems for operational water and wastewater treatment plants that meet exact design specifications and stringent, evolving EPA regulations. Our systems include high-quality components, open-architecture hardware, user-friendly HMIs, extensive documentation, useful analysis for all stakeholders, and comprehensive service.
SCADA systems are essential to successful operations, and there are three critical aspects that organizations with small- and mid-size operations should consider when implementing or considering upgrades to SCADA systems:
1. Consider the Connection
Traditional RTU systems use hard-wired serial connections. These connections require the RTU hardware and software to undergo constant upgrades and software updates for security. Constant updates bog down systems and hinder operation.
However, there is another solution. The DRACS-RTU-3100 is an intelligent SCADA RTU solution for industries like municipal water and wastewater. The DRACS-RTU-3100 uses cloud-based communication for constant communication rather than requiring local hardware and software updates. With seamless communication capabilities through remote smartphone/tablet access, expandable I/O capabilities, and a UL-508A control panel, the DRACS-RTU-3100 is an efficient, scalable, proven solution for an RTU.
2. Use Functional Hardware
Experience with hundreds of water and wastewater applications has given us the insight to understand the most pressing needs of municipalities. Our teams use proven, off-the-shelf hardware to create reliable and efficient process automation and SCADA systems specifically for water and wastewater plants and communication networks. Using COTS hardware keeps costs down and maintenance simple.
The DRACS-RTU-3100 features a rugged NEMA 4x construction and is made with reliable, industrial-grade components for durability in all water and wastewater applications. In addition, it operates with low power consumption and includes a 12+ hour battery backup.
Standard I/O with the platform consists of:
• 20 Digital inputs (dry contact)
• 12 Digital outputs (relay dry contact)
• 4 Analog inputs (4-20 mA DC Inputs at 500 Ω impedance)
• 2 Analog outputs (0 – 10 VDC)
Other options include additional analog input cards, additional analog output cards, alarm indicators, and external high-gain antenna.
The platform also features a standard HMI display when used in conjunction with SMS SCADA. The two platforms combined create a total solution for data communication between a monitored system and the operator. The HMI available with SMS SCADA allows the user to receive data, control, alarms, and status updates through a user-friendly interface.
3. Choose an Experienced Partner
Our applications are already prevalent in the water and wastewater industry, and because of its rugged construction, ease of use, and reliability, the DRACS-RTU-3100 is also ideal for other industries, including:
• Industrial Processes
• Material Handling Systems
• Network Fault Management
• Power Regeneration
At Lord & Company, our clients are the success story – through successful SCADA and automation systems, we provide high-quality, life-saving data that allows clients to impact their communities. We know that changes like a shift to the DRACS-RTU-3100 take time, and our teams work to complete implementations and testing within just weeks. We also support our systems with engineering and field teams after installation.
Diego Machuca, Software Systems Engineer
Lord and company vision is to be the world number one automation systems provider by completing each project on time, under budget and with excellence. In order to accomplish L&C vision, innovation and creativity plays a main key to meet projects expectations and exceed them. Troubleshooting hardware and software often require great amount of effort and time. Some of the tools for troubleshooting either software or hardware are generic for a specific system or a specific brand in the automation industry. These tools in some cases are not enough for a thoroughly troubleshoot process. For this reason Lord and Company has developed internal software tools that helps make this process more effective and less demanding on L&C employees to provide with a better project outcome.
On a current project engineers ran into a situation where a specific decoding software was needed to be able to process large amounts of data and find possible mistakes in the code. It is common on MosCAD networks to transmit data in frames where the programmer has the ability to program such messages. The messages transmitted can be via wireline links, radio link, and local RTU-RTU links. Radio link and local links can be captured via Motorola’s STS tool called Protocol Analyzer and in a more sophisticated wireline link a network sniffer software (Wireshark) can be used to capture all network traffic in a period of time.
Lord and Company has developed software to troubleshoot Motorola MosCAD system network communications. Packets being sent and received can be capture by using a port mirroring device connected to the system network and using any packet sniffing software such as Wireshark. Once the packages have been captured, Lord and Company software can convert and analyze them, providing the programmer with a readable format of the messages in the network. On the first release of the tool the programmer can copy and paste a captured message in hexadecimal format and transformed into a decimal format that is presented in a table format. The table format allows a programmer to understand, verify and validate the contents of the frame.
This software tool was developed using C# and Visual Studio 2013 IDE, it can be used on all Windows operating systems 32 and 64 bit. Programmers at Lord and Company keep improving, innovating and developing internal software tools that will help achieve the company’s vision. Future development of the software tool will have additional features to cross reference a messages with other data pre-defined data, save analyzed data into files, print and some others as needed.
Figure 1. Window with input box and result output table.
Systems Software Engineer, Lord and Company, Inc.
This technical article for Motorola RTU networks using Ethernet links covers the messaging architecture of MDLC over Ethernet. How the MDLC is implemented within the IP structure, how the retry settings affect the messaging and what IP protocol is being used.
In order to document and validate some concepts of MDLC over Ethernet test in a lab environment have been performed. Two ACE 3640 processors with ID’s 1000 and 2000 communicate over Ethernet port (ETH1). An application was loaded into both units to send a message from site 1000 to 2000 using in some cases a TxFrm (transmit frame) and later a SndFrm (send frame) message type. (STS software version 15.50). In order to capture the network traffic between the units a port mirroring device in conjunction with a computer with running Wireshark software network diagnostics software has been used.
Motorola uses UDP (User Datagram Protocol) as opposed to TCP (Transmission Control Protocol). The difference is TCP guarantees the recipient will receive the packets in order by numbering them. The recipient sends messages back to the sender saying it received the messages. When using UDP, packets are just sent to the recipient. The sender won’t wait to make sure the recipient received the packet — it will just continue sending the next packets.
Test 1. The port configuration parameters of both units have default values. When a TxFrm type message is used between sites the following capture is observed on Wireshark:
Fig 1. Wireshark capture between site 1000 and 2000 using TxFrm. Successful communications between sites.
The 3 messages sent back and forth between sites are known as a three way hand shake. It is a three-step method that requires both the client and server to exchange SYN and ACK (acknowledgment) packets before actual data communication begins. The first message (denoted above as No. 1) is the “SYN”, the second message (denoted above as No. 2) is the “ACK” and the third message (denoted above as No. 3) is the actual Data Packet sent from the STS Application. Notice that when both sites are communicating the time delays between each event is the milliseconds order (as expected for a 10/100 communication link)
Test 2. The port configuration parameters of both units have default values. In this test the communication link between sites is failed or disconnected. When site 1000 use a TxFrm to site 2000 the following capture is observed:
Fig 2. Wireshark capture between site 1000 and 2000 using TxFrm. Bad communication between sites
Site 1000 did three attempts of the “SYN” message to the destination site (Site 2000) with a 10 second delay between each try. These results are correlated back to the Advanced Port Settings of the unit:
Fig 3. Partial view of ETH1 Ethernet advance port configuration parameters for sites 1000 and 2000.
The Poll interval parameter is the time delay between retries (10 seconds in this example) and the Maximum number of poll parameter is the number of retries to send a message (3 in this example).
The importance of this parameters is that it allows a programmer to adjust the time delay between retries and the number of “MDLC” retries, without any additional programming at the application level. Configuring this parameter accordingly reduces the possibility of overruns when a message is transmitted. Message overrun is a term we use to describe when the application attempts a retry prior to the MDLC retry attempts expiring, this could cause the transmission buffer of one site to become full and the potentially start dropping messages and affect the performance of the CPU.
Finding the optimum balance between time and number retries and listening time (RTU is not sending messages) can be key to fixing communication issues and streamlining communications. The optimization procedure can be extensive and have multiple iterations before finding the correct balance; considerations to the network structure, speed, data reliability and the particular application will have to be considered to make proper adjustments.
Another important feature of the test was to demonstrate the use of network capture tools such as Wireshark to do diagnostics and help on the parameter settings configuration using reliable data.
On the next article a similar test will be performed using a transmission type SndFrm (Send Frame) between sites. We will show the parameters effect on the sites that communicate using this type of frame.