New control systems have greater communication requirements than in the past. Tighter regulatory restrictions demand that control data must be passed between areas of the plant. Reduced staffing requires that plant information needs to be accessible anywhere in the plant, or sometimes outside of the plant. Advanced graphical interface tools and data logging place a greater demand on the control system’s communications bandwidth. All this combines to make the communications backbone of a Control System a critical component.
Due to the increased demand for speed and reliability, specially designed industrialized hardware and redundant fiber networks were used. The redundant hardware made sure that any one break in a communication path would not restrict the communications. However, this break needed to be corrected before a second issue occurred. The decision was made to use the computer-based SCADA control system to monitor its own health and alarm if a problem was detected.
Hirschmann’s Rail-Switch family were selected from a variety of vendors providing industrialized, managed Ethernet switches because they supported the Ethernet/IP communication protocols used by the control system and could be integrated directly into the system without the need for additional hardware.
The Trihedral VTScada SCADA application was configured to monitor the health of the network switches. A loss of redundancy would be alarmed on by the computer system in the same place that process alarms (high temperature, low pH, etc.) are handled, providing a visual indication, audible indication, historical log, and remote notification by way of phone, email, or text.
Integrating the switch into the system was a three step process: configuration of the switch, implementing the Hirschmann-provided tools in the Rockwell Automation PLC control system, and configuring the computer-based SCADA software to monitor, display, and log alarm conditions.
Step 1: Configure the Switch
Using Hirschmann’s HiDiscovery software tool, the managed switches were configured for redundant fiber-ring topology. In addition to the basic configuration, each switch was configured to enable the Ethernet/IP protocol and the Signal Contact configured to activate on a loss of Redundancy in the ring.
It is important to note that the HiDiscovery software is provided by Hirschmann with all managed switches. However, the other switch configuration tools like the Web-based interface and the Command Line Interface can also be used.
Step 2: Program the PLC
The EDS (Electronic Device Signature) Files and Rockwell Integration tool were downloaded from the Hirschmann website. The EDS files were added to the Rockwell Software Logix5000 programming software using the EDS Hardware Installation Wizard, allowing the Switch to talk to the PLC processor as if it were an IO module. To establish the communications link between the PLC processor and the switch, the switch was simply added to the Ethernet Network tab as an “IO Module”, specifying only the IP Address of the switch and rate of data exchange.
In order to process the information from the switch, an AOI (Add-on Instruction) developed by Hirschmann was imported into the PLC program. This AOI drives the data transfer between the switch and the control processor, extracting the configuration and status information on the health of the switch directly to the PLC program memory variables.
An instruction was added to the PLC program to check the health of the communications between the PLC and the switch using a GSV instruction to monitor the “Entry Status” and generate a communications alarm.
PLC program logic was created to reference the Hirschmann_Switch AOI in order to manipulate information to/from the switch and populate PLC program variables. Information includes port configurations and status, switch temperature, model & serial numbers, diagnostic statistics, and configuration information. Almost all information available from the switch’s Web or Command Line Interface (CLI) are available.
Step 3: Configure the SCADA software
Finally, tags were created in the VTScada SCADA application to alarm on a loss of communications with the switch, or the loss of redundancy of the fiber ring. When a loss of redundancy occurs, the system will indicate the problem audibly and visually in the alarm summary. Visually, the operator can look at a display showing the entire plant communications and quickly identify the problem and begin the process of fixing the fiber ring.
Utilizing a Hirschmann industrial Ethernet solution with Rockwell Automation control hardware and Trihedral’s SCADA monitoring software, Lord and Company was able to provide a cost-effective solution to increase the reliability of the customer’s Water Treatment plant control system. By tightly integrating control network diagnostics with the control system located at the plant the system health can be monitored in real-time using tools with which the plant personnel are familiar. This ensures that potential problems are brought to the attention of people who can act on the information in a timely manner, preventing interruptions in the treatment process or loss of data needed for regulatory reporting.
Customer: City of Reidsville, NC Public Works
Location: Reid Lake Road, Reidsville, NC
Description: 9 MGD capacity plant pulling from Lake Reidsville
· Rockwell Automation CompactLogix Programmable Controller system
· Hirschmann RS20-0800T1T1SAABHH with Industrial Protocols
· Trihedral VTScada SCADA Software running on industrial computers
Lord & Company is a proven systems integrator providing control, monitoring, and information systems for improved, expanded, or new municipal water and wastewater treatments plants, industrial processes, or radio communication systems.
One critical component of these systems is SCADA technology. To understand the importance of a reliable SCADA solution, we will consider three things: current challenges around SCADA systems, key factors in successful systems, and how to choose a reliable solution partner.
Understand Challenges Surrounding SCADA Systems
In the vast networks of municipal water and wastewater plants, industrial operations, or radio systems, we know that SCADA systems are critical to uphold communication. Effective communication systems help organizations reach to remote locations and relieve those remote locations of the burden of requiring additional manpower. However, the time and costs associated with operating typical SCADA systems cause massive headaches for owners looking for seamless control, monitoring, and information systems.
With these challenges in mind, we created SMS (SCADA Made Simple) to offer an easy-to-use HMI solution that brings the benefits of SCADA technology.
Consider Critical Factors for Successful SCADA Solutions
Because SCADA systems manage many processes and withstand much interaction from operators, organizations need a SCADA solution that is secure, robust, and reliable.
SMS SCADA features a robust user interface that allows for information and data communication between monitored systems and the user. It allows operators to send and receive data, and control alarms and status through a user-friendly user interface. It saves time and money by using a cloud-based platform to mitigate the constant hardware and software upgrades required to maintain typical SCADA systems. With a highly secure data center served by a Tier 1 premium bandwidth supplier, Lord & Company has created an integrated solution with dual redundant servers for a safe, robust, reliable HMI solution
SMS SCADA is supported by our 90,000 sq. ft. data center with up to 25,000 servers. High-security data centers with onsite physical and electronic monitoring feature SSAE-16 and HIPPA compliance, and are Safe Harbor Certified.
Choose a Reliable SCADA Solution Provider
At Lord and Company, we are experienced in RF protocols, wireless technology, and the systems relying on your wireless network. Our teams use combined knowledge to help leading companies effectively use wireless communications for dependable and protected SCADA systems with solutions like SMS SCADA. Our CSIA certification, vast network of industry partnerships, and proven experience with large municipal water and wastewater systems like that of Brunswick County, NC, and fire, police, and EMS radio control systems have lead customers to trust working with us at Lord & Company time and again.
Lord And Company, Inc recently completed 3 Weeks of Motorola Toolbox, STS and Wonderware Intouch training for our highly valued customer in the North East. Training was conducted in co-operation with Motorola Solutions, Inc. 12 technical personnel each week for a total of 36 personnel were trained at a convenient facility in Brooklyn, NY. This training was specific to the customer’s System’s hardware, applications and software. Training format included a daily mixture of instructor Presentation and Hands On Lab (8 Labs) with Instructor support.
- Motorola MOSCAD Basics: The course provided knowledge for Toolbox software application backup methods, basic use of and troubleshooting with the Toolbox software, replacement procedures for hardware and reloading of the application programs and configuration files.
- Motorola MOSCAD Advanced Training: Advanced level provided a detailed understanding of the System and Network Configuration, the data tables and ladder programming with procedures for online monitoring of programs and data tables.
- Introduction to Motorola System Tool Suite ”STS” Software for ACE3600 and the differences between STS and Toolbox.
- Motorola STS/ACE3600 Basics: This course provided knowledge for STS software application backup methods, basic use of and troubleshooting with the STS software, replacement procedures for hardware and reloading of the application programs and configuration files.
- Motorola ACE3600 Advanced Training: Advanced level provided a detailed understanding of the System and Network Configuration, the data tables and ladder programming with procedures for online monitoring of programs and data tables.
- Setup and Troubleshooting with a Protocol Analyzer: The course provided the purpose for and instructions of how to setup a protocol analyzer and how to export and interpret the data.
- SCADA System Hardware, Software and Server Basics: The course provided an understanding of the functions of the hardware and software utilized in the system and how it communicates with the Motorola ACE3600 and MOSCAD processors. Includes procedures for rebooting the systems, making system backups and hard drive and data management.
- Wonderware Intouch Basics: This course provided an understanding of the Tag Dictionary db, event logger, Modbus driver configuration for communicating to the FIU/FEP’s, links to the graphics screens and graphics development.
- Lab Setup:
- Twelve (12) Laptops with Motorola System Tool Suite ”STS” Software for ACE3600 , Motorola Toolbox 9.503 Software and Wonderware Intouch software.
- Six (6) MOSCAD RTU’s with 420 Processor, PS and I/O modules and configured and tested for comm to rental PC’s. One RTU per two students.
- Six (6) ACE3600 RTU’s with Processor, PS and I/O modules and configured and tested for comm to rental PC’s. One RTU per two students.
A Big Thank You to the following Team members:
- Kenny Brooks for configuring the Laptops & PLC’s and setting up the lab in our Fort Mill, SC Training Facility. This setup was invaluable to get the training flow synchronized for optimal learning.
- Brooke Morris, for coordinating the shipment of equipment under complex delivery requirements.
- Our Trainers, James Owens (Week 1), Michael Missaggia (Week 2) and Rafael Balderrama (Week 3) for extraordinary team work, combining all the different materials and co-coordinating amongst each other, so everyone could provide the same level of exceptional training.
A Special mention and Thank You to Dana Albina from Motorola Solutions, Inc for providing Training Site and managing logistics in Brooklyn, NY.
Figure 1: Lab Setup for Internal Testing at Lord And Company, Inc Training Facility.
Credit: Kenny Brooks
Figure 2: Reception Area at Training Facility in Brooklyn, NY.
Credit: Dana Albina, Motorola Solutions, Inc
Figure 3: Training Facility in Brooklyn, NY.
Credit: Dana Albina, Motorola Solutions, Inc
Lab Instruction – Slide Samples
Figure 4: Lab 7: STS Setup – Creating New Project Slide.
Figure 5: Lab 7: STS Setup – Compiling Application Sample.
As always, we tried to meet our Goal of Starting and Finishing with Excellence. With a Great Team and Team Work, I believe WE did.
Project Manager, PMP.
Diego Machuca, Software Systems Engineer
Lord and company vision is to be the world number one automation systems provider by completing each project on time, under budget and with excellence. In order to accomplish L&C vision, innovation and creativity plays a main key to meet projects expectations and exceed them. Troubleshooting hardware and software often require great amount of effort and time. Some of the tools for troubleshooting either software or hardware are generic for a specific system or a specific brand in the automation industry. These tools in some cases are not enough for a thoroughly troubleshoot process. For this reason Lord and Company has developed internal software tools that helps make this process more effective and less demanding on L&C employees to provide with a better project outcome.
On a current project engineers ran into a situation where a specific decoding software was needed to be able to process large amounts of data and find possible mistakes in the code. It is common on MosCAD networks to transmit data in frames where the programmer has the ability to program such messages. The messages transmitted can be via wireline links, radio link, and local RTU-RTU links. Radio link and local links can be captured via Motorola’s STS tool called Protocol Analyzer and in a more sophisticated wireline link a network sniffer software (Wireshark) can be used to capture all network traffic in a period of time.
Lord and Company has developed software to troubleshoot Motorola MosCAD system network communications. Packets being sent and received can be capture by using a port mirroring device connected to the system network and using any packet sniffing software such as Wireshark. Once the packages have been captured, Lord and Company software can convert and analyze them, providing the programmer with a readable format of the messages in the network. On the first release of the tool the programmer can copy and paste a captured message in hexadecimal format and transformed into a decimal format that is presented in a table format. The table format allows a programmer to understand, verify and validate the contents of the frame.
This software tool was developed using C# and Visual Studio 2013 IDE, it can be used on all Windows operating systems 32 and 64 bit. Programmers at Lord and Company keep improving, innovating and developing internal software tools that will help achieve the company’s vision. Future development of the software tool will have additional features to cross reference a messages with other data pre-defined data, save analyzed data into files, print and some others as needed.
Figure 1. Window with input box and result output table.
Systems Software Engineer, Lord and Company, Inc.
This technical article for Motorola RTU networks using Ethernet links covers the messaging architecture of MDLC over Ethernet. How the MDLC is implemented within the IP structure, how the retry settings affect the messaging and what IP protocol is being used.
In order to document and validate some concepts of MDLC over Ethernet test in a lab environment have been performed. Two ACE 3640 processors with ID’s 1000 and 2000 communicate over Ethernet port (ETH1). An application was loaded into both units to send a message from site 1000 to 2000 using in some cases a TxFrm (transmit frame) and later a SndFrm (send frame) message type. (STS software version 15.50). In order to capture the network traffic between the units a port mirroring device in conjunction with a computer with running Wireshark software network diagnostics software has been used.
Motorola uses UDP (User Datagram Protocol) as opposed to TCP (Transmission Control Protocol). The difference is TCP guarantees the recipient will receive the packets in order by numbering them. The recipient sends messages back to the sender saying it received the messages. When using UDP, packets are just sent to the recipient. The sender won’t wait to make sure the recipient received the packet — it will just continue sending the next packets.
Test 1. The port configuration parameters of both units have default values. When a TxFrm type message is used between sites the following capture is observed on Wireshark:
Fig 1. Wireshark capture between site 1000 and 2000 using TxFrm. Successful communications between sites.
The 3 messages sent back and forth between sites are known as a three way hand shake. It is a three-step method that requires both the client and server to exchange SYN and ACK (acknowledgment) packets before actual data communication begins. The first message (denoted above as No. 1) is the “SYN”, the second message (denoted above as No. 2) is the “ACK” and the third message (denoted above as No. 3) is the actual Data Packet sent from the STS Application. Notice that when both sites are communicating the time delays between each event is the milliseconds order (as expected for a 10/100 communication link)
Test 2. The port configuration parameters of both units have default values. In this test the communication link between sites is failed or disconnected. When site 1000 use a TxFrm to site 2000 the following capture is observed:
Fig 2. Wireshark capture between site 1000 and 2000 using TxFrm. Bad communication between sites
Site 1000 did three attempts of the “SYN” message to the destination site (Site 2000) with a 10 second delay between each try. These results are correlated back to the Advanced Port Settings of the unit:
Fig 3. Partial view of ETH1 Ethernet advance port configuration parameters for sites 1000 and 2000.
The Poll interval parameter is the time delay between retries (10 seconds in this example) and the Maximum number of poll parameter is the number of retries to send a message (3 in this example).
The importance of this parameters is that it allows a programmer to adjust the time delay between retries and the number of “MDLC” retries, without any additional programming at the application level. Configuring this parameter accordingly reduces the possibility of overruns when a message is transmitted. Message overrun is a term we use to describe when the application attempts a retry prior to the MDLC retry attempts expiring, this could cause the transmission buffer of one site to become full and the potentially start dropping messages and affect the performance of the CPU.
Finding the optimum balance between time and number retries and listening time (RTU is not sending messages) can be key to fixing communication issues and streamlining communications. The optimization procedure can be extensive and have multiple iterations before finding the correct balance; considerations to the network structure, speed, data reliability and the particular application will have to be considered to make proper adjustments.
Another important feature of the test was to demonstrate the use of network capture tools such as Wireshark to do diagnostics and help on the parameter settings configuration using reliable data.
On the next article a similar test will be performed using a transmission type SndFrm (Send Frame) between sites. We will show the parameters effect on the sites that communicate using this type of frame.
I am proud to say that I have been working at Lord & Company for almost 20 years. I started with the company in May of 1996. One of the things that impressed me most about the company was their generosity. Generosity is an area of my life that I have struggled with since I can remember as a small boy growing up in a rural area of Carroll County, Virginia.
We all like to exaggerate about how it was when we were kids. You know the old stories of walking to school uphill in the snow both ways. I always had food to eat, clothing and anything that I needed. I did not grow up with very many items that I may have wanted. I saw others with much more and I wanted to be like them. At some point in my early childhood I made a decision that I was willing to work hard but I was going to get those things that I wanted. Unfortunately, I decided to become a taker.
As the years have passed by I have begun to see that it is not about Taking as much as it is about Giving. The little boy inside of me has finally realized that he wants to become a Giver and not a Taker. This process started when I repented of my sins and ask Jesus Christ to come into my life on November 29, 1992. That was the single Greatest day of my life and since that day it has been a process of change.
I am impressed with so many of my co-workers that give their time, talent and resources outside of work each week. Some of them coach Special Olympics, others teach or train youth, while some raise money for charities or food for the hungry. They come together for special events like Operation Christmas Shoebox and go over our goal of 200 boxes. There too many to mention in this short article.
Lord & Company is a generous company that cares about our community and those less fortunate. I have been blessed to see giving in action firsthand over the past 20 years. Yes we have the most talented professional staff that I have ever worked with in my career but they are so much more. They have demonstrated to me how to become a Giver and not a Taker.
You can ask anyone that knows me and they will tell you that I appreciate a job well done. Well done Lord & Company Team. Thank you for demonstrating volunteer work and for helping me to start opening up my hand and become that giver that God has always wanted me to become. Lord & Company is a generous company made up of generous talented people that care about others.
Keep on making a difference in my life and those around me!
Tom Carico – PMP
Lord & Company, Inc.
On Friday October 9th Team L&C had the opportunity to attend the Live2Lead conference. The speakers presented and shared key leadership tools to the participants. This event was broadcast to more than 40,000 people in USA and around the world.
John Maxwell and others, delivered passionate messages on how to improve our lives and the lives of people we work with. The main theme was developing a positive attitude in all circumstances as leaders.
The presentation emphasized that consistency and passion for our work produces positive working environments. Implementing new action plans to lead a team to succeed was also an emphasis.
I have been a recreational tennis players for many years. I had the opportunity to hear the word consistency in a coach program I attended this year. The key to win a match is not only to hit the tennis ball hard but being able to hit it inside the court and over net every time. This is as many people refer, “to be consistent”. In our work environment we need to develop a similar attitude to be the best in our trade and to set an example for our team members.
I also found interesting the concept of differences in “Front end thinking” and “Back end thinking”. In front end thinking we plan, set goals and priorities for good outcomes instead of looking at the back end results of unplanned activities.
L&C team leaders were impressed with the quality and content of the message of the speakers. As a team leader I look forward to using the concepts within the Lord & Company. This will lead us to have continual growth and we can become a better organization to our society.
We want to thank Keith Harris of Fellowship Management and Lord & Company for coordinating and promoting this event in Rock hill, SC.
By Brooke Morris, Design Team Member
Operation Christmas Child is a project of Samaritans Purse that partners with Churches in foreign countries to spread the Gospel to boys and girls. OCC presents a shoebox full of gifts and essentials to the children, and many disciple through a Bible study course. Each year Churches and companies across the region collect and pack shoeboxes to expand and grow the spread the word of Jesus Christ.
The method of filling a shoebox is simple. You can use an adult size shoebox or get a preformed box from the ministry. The shoebox, made up of a “Wow item”, a special gift for the child such as soccer ball with pump, stuffed animals, dolls, toy cars and trucks. Also the box should contain school supplies, non-liquid hygiene items, accessories, crafts and other fun toys or activity books. There are 3 age levels and you can choose the gender of the child also. Girls 2-4, Girls 5-9, Girls 10-14 & Boys 2-4, Boys 5-9, Boys 10-14. There is a suggested donation of $7 to cover the shipping of the shoebox. You can even track your box online by completing the tracking label and submitting your $7 donation for shipping.
Lord & Company, Inc. started participating in Operation Christmas Child in 2013. Our first year we collected about 25 boxes. In 2014 we encouraged our company to take part more and set and met a goal of 60 boxes.
For 2015 we wanted to push further and get even more people involved in the project. Our goal is 200 boxes. To help reach this goal we had a company meeting to explain exactly what Operation Christmas Child is all about and showed our teams how to pack a shoebox. We have planned a packing party for November 6th and have encouraged our employees to get their kids and family involved. I know personally that my 15 year old son and 4 year old grandson love to help shop for the boxes. They are learning to give to others. These are kids that don’t have the everyday necessities that are often taken for granted by children in the US.
By adding something new each year, we hope to gain more interest and thus collect more boxes. To maximize our collection for 2015 we are doing a Penny Drive. Each L&C team gets a collection pale and challenged to see who could bring in the most change by November 6th for our packing party. The change collection will buy extra items needed when we pack the boxes.
Our final collection day is November 13th. We will pray over the boxes for God’s will and the Love of Jesus to spread and shared through the children that receive this special gift. Our boxes will then shipped to Samaritan’s Purse and distributed during December.
I have been working in the Engineering Field for about 30 years now and one thing has not changed. We need to be able to do more work in less hours. I have read books on time management and went to seminars to help me develop time management skills. I also know that everyone is different but I wanted to share a few items that has worked for me and helped me with my time management. I hope that you will find some items that will help you with your time management.
Be aware and on the outlook for the small things that distract you from your work each day. You may want to check your emails 2 or 3 times a day but do not leave them up in front of you at all times. If you stop what you are doing to read each email as it appears on your screen this will take up half of your workday. Impromptu meetings can also be a leading cause of disruptions in your work day. Request that meetings have a schedule with an agenda so that it minimizes everyone’s time involved. I know that meetings are necessary to get work done in a team environment but planned and scheduled meeting are far more productive. I enjoy attending and being part of productive meetings but I do not recall a productive meeting that I attended without an agenda.
Have a written plan for each day. This helps me stay focused on the work that has you’ve decided to be a priority for the work day. This helps me stay on track each day and helps me to see my daily goals. I typically use my tablet for this purpose.
Keep a daily progress report. How many times have I worked 9, 10 or more hours in one day and left work feeling like I was not productive? The answer is way too many times. Keeping a daily progress report helps me focus on each day’s successes even when I worked on items that were not planned. Yes, no matter how much planning you do there will be days that outside factors force you to change your priority list for the day and you work on other items. Keeping a daily progress report helps me remember to focus on the work items that I completed for the day and not the items that I did not get accomplished.
Answer Phone calls. If you are not deep in the middle of a task you should go ahead and answer that phone call. I know we all get too many phone calls that we may consider non-productive but it may take less time to take the call than to return the phone call later. Remember that we all want someone to answer our phone calls so we should do the same. If you have voice messages, set aside a couple of times a day to return phone calls. I try to return calls at 10 AM and at 3 PM each day.
A Good night’s sleep. Each day starts so much better if you get a good night’s sleep. I have found that I work better with 7 hours sleep each night. I can make it on less sleep but it catches up with me by the end of the week. I know that coffee is my friend in the morning since I am not a morning person but coffee will not make me productive all day long. Sleep prepares my mind and body for each work day. Do not underestimate the importance of a good night’s sleep.
Multi-tasking. I have tried to multi-task and I have found myself making too many costly mistakes. I work on many projects and I wear many hats each day so this is a daily thorn in my side. I have my work schedule, my home family schedule, my church schedule, my volunteer schedule, my reading schedule and so on … You get the picture. We are all busy and if we are not careful we try to focus on too many things at one time and we are not effective at any of them. Concentrate on one item at a time and complete it. I try to keep one schedule for work and home with alert reminders to help me remember all the items that I want to do and schedule. I often hear people say that I wish I could remember this or that thing tomorrow. I typically say if we could only have someone invent something that was small and easy to carry that we could schedule reminders to alert us tomorrow? It is too bad that you do not have such a device like this that could help you. Use your Smart Phone to help you and remind you of anything and everything that is important to you.
One of our greatest resources that we have is time. No matter rich, poor, young, older God gave us all the same 24 hours each day. Make a plan and use your time wisely. I hope that some of the things that I have learned and shared on time management will help you focus and become more productive each day. I continue to work on improving my time management skills each day. Speaking of time management; the phone is ringing and it is time for me to get back to work.
Tom Carico – PMP
Lord & Company, Inc.