Blog

Monitoring Control System Network Health

 

The Challenge:

New control systems have greater communication requirements than in the past. Tighter regulatory restrictions demand that control data must be passed between areas of the plant. Reduced staffing requires that plant information needs to be accessible anywhere in the plant, or sometimes outside of the plant. Advanced graphical interface tools and data logging place a greater demand on the control system’s communications bandwidth. All this combines to make the communications backbone of a Control System a critical component.
Due to the increased demand for speed and reliability, specially designed industrialized hardware and redundant fiber networks were used. The redundant hardware made sure that any one break in a communication path would not restrict the communications. However, this break needed to be corrected before a second issue occurred. The decision was made to use the computer-based SCADA control system to monitor its own health and alarm if a problem was detected.
Hirschmann’s Rail-Switch family were selected from a variety of vendors providing industrialized, managed Ethernet switches because they supported the Ethernet/IP communication protocols used by the control system and could be integrated directly into the system without the need for additional hardware.
The Trihedral VTScada SCADA application was configured to monitor the health of the network switches. A loss of redundancy would be alarmed on by the computer system in the same place that process alarms (high temperature, low pH, etc.) are handled, providing a visual indication, audible indication, historical log, and remote notification by way of phone, email, or text.

 

 

The Solution:
Integrating the switch into the system was a three step process: configuration of the switch, implementing the Hirschmann-provided tools in the Rockwell Automation PLC control system, and configuring the computer-based SCADA software to monitor, display, and log alarm conditions.

Step 1: Configure the Switch
Using Hirschmann’s HiDiscovery software tool, the managed switches were configured for redundant fiber-ring topology. In addition to the basic configuration, each switch was configured to enable the Ethernet/IP protocol and the Signal Contact configured to activate on a loss of Redundancy in the ring.
It is important to note that the HiDiscovery software is provided by Hirschmann with all managed switches. However, the other switch configuration tools like the Web-based interface and the Command Line Interface can also be used.

Step 2: Program the PLC
The EDS (Electronic Device Signature) Files and Rockwell Integration tool were downloaded from the Hirschmann website. The EDS files were added to the Rockwell Software Logix5000 programming software using the EDS Hardware Installation Wizard, allowing the Switch to talk to the PLC processor as if it were an IO module. To establish the communications link between the PLC processor and the switch, the switch was simply added to the Ethernet Network tab as an “IO Module”, specifying only the IP Address of the switch and rate of data exchange.
In order to process the information from the switch, an AOI (Add-on Instruction) developed by Hirschmann was imported into the PLC program. This AOI drives the data transfer between the switch and the control processor, extracting the configuration and status information on the health of the switch directly to the PLC program memory variables.
An instruction was added to the PLC program to check the health of the communications between the PLC and the switch using a GSV instruction to monitor the “Entry Status” and generate a communications alarm.

PLC program logic was created to reference the Hirschmann_Switch AOI in order to manipulate information to/from the switch and populate PLC program variables. Information includes port configurations and status, switch temperature, model & serial numbers, diagnostic statistics, and configuration information. Almost all information available from the switch’s Web or Command Line Interface (CLI) are available.

 

 

 

Step 3: Configure the SCADA software


Finally, tags were created in the VTScada SCADA application to alarm on a loss of communications with the switch, or the loss of redundancy of the fiber ring. When a loss of redundancy occurs, the system will indicate the problem audibly and visually in the alarm summary. Visually, the operator can look at a display showing the entire plant communications and quickly identify the problem and begin the process of fixing the fiber ring.

 

 

Conclusion: 

Utilizing a Hirschmann industrial Ethernet solution with Rockwell Automation control hardware and Trihedral’s SCADA monitoring software, Lord and Company was able to provide a cost-effective solution to increase the reliability of the customer’s Water Treatment plant control system. By tightly integrating control network diagnostics with the control system located at the plant the system health can be monitored in real-time using tools with which the plant personnel are familiar. This ensures that potential problems are brought to the attention of people who can act on the information in a timely manner, preventing interruptions in the treatment process or loss of data needed for regulatory reporting.

 

James Ownes

 

Customer:      City of Reidsville, NC Public Works

Location:       Reid Lake Road, Reidsville, NC

Description:   9 MGD capacity plant pulling from Lake Reidsville

 

System Components:

·         Rockwell Automation CompactLogix Programmable Controller system

·         Hirschmann RS20-0800T1T1SAABHH with Industrial Protocols

·         Trihedral VTScada SCADA Software running on industrial computers

Modernize Your Monitoring Equipment With the 2+2(s) Compact Cellular Transceiver

The new Lord & Company 2+2(s) LC 300 is a compact cellular transceiver for monitoring equipment that sends real-time alarms and status updates via an online portal. There’s no need to wait for updates via an auto-dialer or other outdated technology – the transceiver uses cellular signals to instantly send information.

 

Figure 1. The 2+2(s) transceiver

 

In addition to alarming on status changes, the system keeps track of on-time and digital input cycles to help when monitoring pump, fan, or compressor run-times and duty cycles. Its analog inputs can alarm on high or low conditions and are useful for monitoring temperatures, pressures, tank levels or flow rates. Additionally, the unit monitors input power voltage and provides a battery backup, enabling it to alarm on the main power voltage level or failure and continue operating without main power.

 

The 2+2(s) sends real-time data to an online portal, displaying the location and operating status for your monitored equipment. If desired, you can set up the portal to immediately send updates/alarms via SMS text, messages, and emails. It permanently logs every event, providing valuable historical data that you can analyze via powerful built-in reporting features. View the video demo below for a walk-through of the hardware and online portal, which demonstrate the breadth of the 2+2’s capabilities.

 

 

If you’re ready to learn how the 2+2(s) can meet your specific needs, contact us today.

Reclaim your Time with an HMI Software Hosting Solution: SMS SCADA

Lord & Company is a proven systems integrator providing control, monitoring, and information systems for improved, expanded, or new municipal water and wastewater treatments plants, industrial processes, or radio communication systems.

 

One critical component of these systems is SCADA technology. To understand the importance of a reliable SCADA solution, we will consider three things: current challenges around SCADA systems, key factors in successful systems, and how to choose a reliable solution partner.

 

Understand Challenges Surrounding SCADA Systems

In the vast networks of municipal water and wastewater plants, industrial operations, or radio systems, we know that SCADA systems are critical to uphold communication. Effective communication systems help organizations reach to remote locations and relieve those remote locations of the burden of requiring additional manpower. However, the time and costs associated with operating typical SCADA systems cause massive headaches for owners looking for seamless control, monitoring, and information systems.

 

With these challenges in mind, we created SMS (SCADA Made Simple) to offer an easy-to-use HMI solution that brings the benefits of SCADA technology.

 

Consider Critical Factors for Successful SCADA Solutions

Because SCADA systems manage many processes and withstand much interaction from operators, organizations need a SCADA solution that is secure, robust, and reliable.

 

SMS SCADA features a robust user interface that allows for information and data communication between monitored systems and the user. It allows operators to send and receive data, and control alarms and status through a user-friendly user interface. It saves time and money by using a cloud-based platform to mitigate the constant hardware and software upgrades required to maintain typical SCADA systems. With a highly secure data center served by a Tier 1 premium bandwidth supplier, Lord & Company has created an integrated solution with dual redundant servers for a safe, robust, reliable HMI solution

.

 

SMS SCADA is supported by our 90,000 sq. ft. data center with up to 25,000 servers. High-security data centers with onsite physical and electronic monitoring feature SSAE-16 and HIPPA compliance, and are Safe Harbor Certified.

 

Choose a Reliable SCADA Solution Provider

At Lord and Company, we are experienced in RF protocols, wireless technology, and the systems relying on your wireless network. Our teams use combined knowledge to help leading companies effectively use wireless communications for dependable and protected SCADA systems with solutions like SMS SCADA. Our CSIA certification, vast network of industry partnerships, and proven experience with large municipal water and wastewater systems like that of Brunswick County, NC, and fire, police, and EMS radio control systems have lead customers to trust working with us at Lord & Company time and again.

 

 

Save Time and Money with an Intelligent SCADA RTU: DRACS-RTU-3100

At Lord & Company, we partner with clients to create and sustain industrial automation systems for operational water and wastewater treatment plants that meet exact design specifications and stringent, evolving EPA regulations. Our systems include high-quality components, open-architecture hardware, user-friendly HMIs, extensive documentation, useful analysis for all stakeholders, and comprehensive service.

 

SCADA systems are essential to successful operations, and there are three critical aspects that organizations with small- and mid-size operations should consider when implementing or considering upgrades to SCADA systems:

 
1. Consider the Connection
Traditional RTU systems use hard-wired serial connections. These connections require the RTU hardware and software to undergo constant upgrades and software updates for security. Constant updates bog down systems and hinder operation.

However, there is another solution. The DRACS-RTU-3100 is an intelligent SCADA RTU solution for industries like municipal water and wastewater. The DRACS-RTU-3100 uses cloud-based communication for constant communication rather than requiring local hardware and software updates. With seamless communication capabilities through remote smartphone/tablet access, expandable I/O capabilities, and a UL-508A control panel, the DRACS-RTU-3100 is an efficient, scalable, proven solution for an RTU.

 

2. Use Functional Hardware
Experience with hundreds of water and wastewater applications has given us the insight to understand the most pressing needs of municipalities. Our teams use proven, off-the-shelf hardware to create reliable and efficient process automation and SCADA systems specifically for water and wastewater plants and communication networks. Using COTS hardware keeps costs down and maintenance simple.

dracs-3100

The DRACS-RTU-3100 features a rugged NEMA 4x construction and is made with reliable, industrial-grade components for durability in all water and wastewater applications. In addition, it operates with low power consumption and includes a 12+ hour battery backup.

Standard I/O with the platform consists of:
• 20 Digital inputs (dry contact)
• 12 Digital outputs (relay dry contact)
• 4 Analog inputs (4-20 mA DC Inputs at 500 Ω impedance)
• 2 Analog outputs (0 – 10 VDC)

Other options include additional analog input cards, additional analog output cards, alarm indicators, and external high-gain antenna.

The platform also features a standard HMI display when used in conjunction with SMS SCADA. The two platforms combined create a total solution for data communication between a monitored system and the operator. The HMI available with SMS SCADA allows the user to receive data, control, alarms, and status updates through a user-friendly interface.

 

3. Choose an Experienced Partner
Our applications are already prevalent in the water and wastewater industry, and because of its rugged construction, ease of use, and reliability, the DRACS-RTU-3100 is also ideal for other industries, including:
• Industrial Processes
• Material Handling Systems
• Mining
• Network Fault Management
• Power Regeneration

 

At Lord & Company, our clients are the success story – through successful SCADA and automation systems, we provide high-quality, life-saving data that allows clients to impact their communities. We know that changes like a shift to the DRACS-RTU-3100 take time, and our teams work to complete implementations and testing within just weeks. We also support our systems with engineering and field teams after installation.

Developing Software Tools to Troubleshoot Networks

Diego Machuca, Software Systems Engineer

Lord and company vision is to be the world number one automation systems provider by completing each project on time, under budget and with excellence. In order to accomplish L&C vision, innovation and creativity plays a main key to meet projects expectations and exceed them. Troubleshooting hardware and software often require great amount of effort and time. Some of the tools for troubleshooting either software or hardware are generic for a specific system or a specific brand in the automation industry. These tools in some cases are not enough for a thoroughly troubleshoot process. For this reason Lord and Company has developed internal software tools that helps make this process more effective and less demanding on L&C employees to provide with a better project outcome.

 

On a current project engineers ran into a situation where a specific decoding software was needed to be able to process large amounts of data and find possible mistakes in the code. It is common on MosCAD networks to transmit data in frames where the programmer has the ability to program such messages. The messages transmitted can be via wireline links, radio link, and local RTU-RTU links. Radio link and local links can be captured via Motorola’s STS tool called Protocol Analyzer and in a more sophisticated wireline link a network sniffer software (Wireshark) can be used to capture all network traffic in a period of time.

 

Lord and Company has developed software to troubleshoot Motorola MosCAD system network communications. Packets being sent and received can be capture by using a port mirroring device connected to the system network and using any packet sniffing software such as Wireshark. Once the packages have been captured, Lord and Company software can convert and analyze them, providing the programmer with a readable format of the messages in the network. On the first release of the tool the programmer can copy and paste a captured message in hexadecimal format and transformed into a decimal format that is presented in a table format. The table format allows a programmer to understand, verify and validate the contents of the frame.

 

This software tool was developed using C# and Visual Studio 2013 IDE, it can be used on all Windows operating systems 32 and 64 bit. Programmers at Lord and Company keep improving, innovating and developing internal software tools that will help achieve the company’s vision. Future development of the software tool will have additional features to cross reference a messages with other data pre-defined data, save analyzed data into files, print and some others as needed.

 

machuca1

Figure 1. Window with input box and result output table.

Optimizing Motorola MDLC Over IP

Chris Phelps
Systems Software Engineer, Lord and Company, Inc.

This technical article for Motorola RTU networks using Ethernet links covers the messaging architecture of MDLC over Ethernet. How the MDLC is implemented within the IP structure, how the retry settings affect the messaging and what IP protocol is being used.
In order to document and validate some concepts of MDLC over Ethernet test in a lab environment have been performed. Two ACE 3640 processors with ID’s 1000 and 2000 communicate over Ethernet port (ETH1). An application was loaded into both units to send a message from site 1000 to 2000 using in some cases a TxFrm (transmit frame) and later a SndFrm (send frame) message type. (STS software version 15.50). In order to capture the network traffic between the units a port mirroring device in conjunction with a computer with running Wireshark software network diagnostics software has been used.
Motorola uses UDP (User Datagram Protocol) as opposed to TCP (Transmission Control Protocol). The difference is TCP guarantees the recipient will receive the packets in order by numbering them. The recipient sends messages back to the sender saying it received the messages. When using UDP, packets are just sent to the recipient. The sender won’t wait to make sure the recipient received the packet — it will just continue sending the next packets.
Test 1. The port configuration parameters of both units have default values. When a TxFrm type message is used between sites the following capture is observed on Wireshark:

Fig 1. Wireshark capture between site 1000 and 2000 using TxFrm. Successful communications between sites.

The 3 messages sent back and forth between sites are known as a three way hand shake. It is a three-step method that requires both the client and server to exchange SYN and ACK (acknowledgment) packets before actual data communication begins. The first message (denoted above as No. 1) is the “SYN”, the second message (denoted above as No. 2) is the “ACK” and the third message (denoted above as No. 3) is the actual Data Packet sent from the STS Application. Notice that when both sites are communicating the time delays between each event is the milliseconds order (as expected for a 10/100 communication link)
Test 2. The port configuration parameters of both units have default values. In this test the communication link between sites is failed or disconnected. When site 1000 use a TxFrm to site 2000 the following capture is observed:

Fig 2. Wireshark capture between site 1000 and 2000 using TxFrm.  Bad communication between sites

Site 1000 did three attempts of the “SYN” message to the destination site (Site 2000) with a 10 second delay between each try. These results are correlated back to the Advanced Port Settings of the unit:

 Fig 3. Partial view of ETH1 Ethernet advance port configuration parameters for sites 1000 and 2000.

The Poll interval parameter is the time delay between retries (10 seconds in this example) and the Maximum number of poll parameter is the number of retries to send a message (3 in this example).

The importance of this parameters is that it allows a programmer to adjust the time delay between retries and the number of “MDLC” retries, without any additional programming at the application level.  Configuring this parameter accordingly reduces the possibility of overruns when a message is transmitted.  Message overrun is a term we use to describe when the application attempts a retry prior to the MDLC retry attempts expiring, this could cause the transmission buffer of one site to become full and the potentially start dropping messages and affect the performance of the CPU.

Finding the optimum balance between time and number retries and listening time (RTU is not sending messages) can be key to fixing communication issues and streamlining communications. The optimization procedure can be extensive and have multiple iterations before finding the correct balance; considerations to the network structure, speed, data reliability and the particular application will have to be considered to make proper adjustments.

Another important feature of the test was to demonstrate the use of network capture tools such as Wireshark to do diagnostics and help on the parameter settings configuration using reliable data.

On the next article a similar test will be performed using a transmission type SndFrm (Send Frame) between sites.  We will show the parameters effect on the sites that communicate using this type of frame.

Giving Versus Taking

Give

I am proud to say that I have been working at Lord & Company for almost 20 years.  I started with the company in May of 1996.  One of the things that impressed me most about the company was their generosity.  Generosity is an area of my life that I have struggled with since I can remember as a small boy growing up in a rural area of Carroll County, Virginia.

 

We all like to exaggerate about how it was when we were kids.  You know the old stories of walking to school uphill in the snow both ways.  I always had food to eat, clothing and anything that I needed.  I did not grow up with very many items that I may have wanted.  I saw others with much more and I wanted to be like them.  At some point in my early childhood I made a decision that I was willing to work hard but I was going to get those things that I wanted.  Unfortunately, I decided to become a taker.

 

As the years have passed by I have begun to see that it is not about Taking as much as it is about Giving.  The little boy inside of me has finally realized that he wants to become a Giver and not a Taker.  This process started when I repented of my sins and ask Jesus Christ to come into my life on November 29, 1992.  That was the single Greatest day of my life and since that day it has been a process of change.

Helping Hands

I am impressed with so many of my co-workers that give their time, talent and resources outside of work each week.  Some of them coach Special Olympics, others teach or train youth, while some raise money for charities or food for the hungry.  They come together for special events like Operation Christmas Shoebox and go over our goal of 200 boxes.  There too many to mention in this short article.

 

Lord & Company is a generous company that cares about our community and those less fortunate.  I have been blessed to see giving in action firsthand over the past 20 years.  Yes we have the most talented professional staff that I have ever worked with in my career but they are so much more.  They have demonstrated to me how to become a Giver and not a Taker.

Helping

You can ask anyone that knows me and they will tell you that I appreciate a job well done.  Well done Lord & Company Team.  Thank you for demonstrating volunteer work and for helping me to start opening up my hand and become that giver that God has always wanted me to become.  Lord & Company is a generous company made up of generous talented people that care about others.

Keep on making a difference in my life and those around me!     

#GivernotaTaker    

 

Tom Carico – PMP

Lord & Company, Inc.

The Places You Will See; Gary Mintz

curve-2-final2-990As I drive around the southeast, (NC, SC, GA, TN, VA & Washington, DC), I can tell those who are traveling with me that Lord and Company, Inc. installed that antenna on that tower or water tank you see in the distance.  Or this is the place where we installed PLCs and wrote programs and created graphics to let the operators know the status is of their systems.  Or it might be that we are gathering data via radio to know when to turn on pumps to fill the water tanks we are passing by on the road.  It makes me feel good to know that I have friends in these places.  These are people who I feel I could call on if I needed assistance in that area.  There is some comfort in knowing wherever you go, you could be near someone you know.

 

I will never forget the experience on Tuesday morning, September 11, 2001.  I was in Lockhart, SC.  This was one of my first TLI projects – a small control panel with a large led display that could be turned on remotely to show the customer the amount of water in their tank.  Saves the customer time now that they no longer have to unlock the gate, go in to read a value from inside the location.  I was here when the World Trade Center Towers were destroyed.  My daughter called to ask if I was alright, and I told her that where I was, no one would waste time bombing here because I’m in the middle of a large field.  I am okay.  How about you?

 

The historic city of New Bern, NC has memories for me as well.  This project had several PLCs, SCADA and fiber optics.  My first time traveling to New Bern was with Tim, one of our programmers.  Tim and I were able to discuss the project and I knew he had a good grasp on what was to be done.  We met the folks at New Bern, discussed our plans and they worked with us during the installation.  We hired someone nearby to install the fiber optics and test the installation for us.  I learned valuable information about fiber optic installations.

 

I pass near Lumberton, NC on my way to the beach.  I did PLC work there before I worked for Lord and Company and when I was given the opportunity to replace the Square D PLCs with Modicon PLCs, I was delighted.  Upon my first trip there, I was taken aback when they said this ODS panel does not work.  I said if it is not working and we are going to replace it, let’s be sure we get it working like you want it to before I write the new program.  On my next trip I asked what did not work.  I found the outputs quickly and noted the logic when called to run the pumps was not turning on the output.  There was a timer in the program that had no entry for the amount of time delay.  I entered a value into the reserved location and voila.  The logic showed the output was energized.  But, wait, the pump still was not coming on.  Let’s check the output fuse.  Sure enough, it was blown.  A replacement fuse was installed and this time, the ODS pumps worked.  Now the fun part – reprogram what was done many years previously.  Any time I am passing nearby, I call and ask, “How are you?  Is there anything I can do for you?”  I know the folks there really don’t need me anymore because they have very capable people running the plant.  However, I still like to keep in touch.

 

Upon my traveling home, about half way, I pass through the small town of Salisbury, NC.  Years prior, I called on these folks in a sales role for an electrical manufacturer.  I was delighted when appointed to handle their account with Lord and Company.  Salisbury was adding a new filtration system furnished by another integrator.  The new system must run simultaneously with their existing system.  They needed to be able to switch from the old system to the new system without losing data in their SCADA system.  When the day of the changeover occurred, there was no clue that anything had ever changed.  It was a successful merger of the two systems.  I had some really smart programmers working on the SCADA portion.  All I had to do was figure out what the previous systems integrator had programmed and understand their logic so I could advise our people what to expect, where to look for important data and they were well on their way to making all items appear as desired.

 

My family had a small trailer across the Intercoastal Waterway from Oak Island, NC.  When I was assigned a project in Supply, NC within seven miles of that trailer, I felt like I was home.  Previously I did some PLC work for Brunswick County.  On this project Lord and Company supplied nine control panels networked via fiber optics all around their water plant.  My New Bern experience helped.  Our service technicians were now able to make the fiber optic terminations.  Lord and Company has grown into a successful systems integrator.  I have close friends in Brunswick County and when I travel to the beach, they are remembered.  When hurricanes cause them problems, I call and ask how they came through.  God has blessed them as so far no damage has been done to their homes or lives as a result of hurricanes.

 

One last place is Washington.  Did I say, NC or DC?  I could mention both because I have been to each with Lord and Company.  I am into my 8th year doing something with the AOC.  These folks are quietly demanding and understanding about the construction process.  They are constantly meeting to be sure all parties are on the same page.  They handle questions and provide specific answers to the questions.  From the control room operators to the board room, the folks are really good to work with.  The operators are willing to learn what the new controls will be doing.  They know they have to run the system after we are gone and they get involved to make things work right.  They don’t get in the way, mind you, they just help when asked.  From coal fired boiler additions to boiler feedwater pumps to 5,000 ton air conditioning chillers all networked via fiber optics and coming to a central DCS system, Lord and Company has been there since 2007 to provide them the controls to run the pumps and equipment.

 

Also, in the Washington, DC area is the DC Blue Plains Enhanced Wastewater Treatment Plant.  Lord and Company is working with Ulliman Schutte Construction, LLC to provide/modify control panels, add flow meters and various other instruments to provide a distributive control system at one of the largest wastewater treatment plants in the United States.

 

When I travel, I see places where I have had an opportunity to make a difference.  It is a treat to ride down the highway knowing that you were somewhat responsible for the way that equipment operates.  It gives me a warm fuzzy feeling that I was able to help someone have a better life.  I have friends in these towns and there is not enough space to write about what took place at each location but it brings back memories to me – these places you will see.

Photo by James Clear

Live2Lead 2015, by Rafael Balderrama

L2L Pic
On Friday October 9th Team L&C had the opportunity to attend the Live2Lead conference. The speakers presented and shared key leadership tools to the participants. This event was broadcast to more than 40,000 people in USA and around the world.

John Maxwell and others, delivered passionate messages on how to improve our lives and the lives of people we work with. The main theme was developing a positive attitude in all circumstances as leaders.

The presentation emphasized that consistency and passion for our work produces positive working environments.  Implementing new action plans to lead a team to succeed was also an emphasis.

I have been a recreational tennis players for many years. I had the opportunity to hear the word consistency in a coach program I attended this year.  The key to win a match is not only to hit the tennis ball hard but being able to hit it inside the court and over net every time. This is as many people refer, “to be consistent”.  In our work environment we need to develop a similar attitude to be the best in our trade and to set an example for our team members.

I also found interesting the concept of differences in “Front end thinking” and “Back end thinking”.  In front end thinking we plan, set goals and priorities for good outcomes instead of looking at the back end results of unplanned activities.

L&C team leaders were impressed with the quality and content of the message of the speakers.  As a team leader I look forward to using the concepts within the Lord & Company. This will lead us to have continual growth and we can become a better organization to our society.

We want to thank Keith Harris of Fellowship Management and Lord & Company for coordinating and promoting this event in Rock hill, SC.