Archive for October 2016

Developing Software Tools to Troubleshoot Networks

Diego Machuca, Software Systems Engineer

Lord and company vision is to be the world number one automation systems provider by completing each project on time, under budget and with excellence. In order to accomplish L&C vision, innovation and creativity plays a main key to meet projects expectations and exceed them. Troubleshooting hardware and software often require great amount of effort and time. Some of the tools for troubleshooting either software or hardware are generic for a specific system or a specific brand in the automation industry. These tools in some cases are not enough for a thoroughly troubleshoot process. For this reason Lord and Company has developed internal software tools that helps make this process more effective and less demanding on L&C employees to provide with a better project outcome.

 

On a current project engineers ran into a situation where a specific decoding software was needed to be able to process large amounts of data and find possible mistakes in the code. It is common on MosCAD networks to transmit data in frames where the programmer has the ability to program such messages. The messages transmitted can be via wireline links, radio link, and local RTU-RTU links. Radio link and local links can be captured via Motorola’s STS tool called Protocol Analyzer and in a more sophisticated wireline link a network sniffer software (Wireshark) can be used to capture all network traffic in a period of time.

 

Lord and Company has developed software to troubleshoot Motorola MosCAD system network communications. Packets being sent and received can be capture by using a port mirroring device connected to the system network and using any packet sniffing software such as Wireshark. Once the packages have been captured, Lord and Company software can convert and analyze them, providing the programmer with a readable format of the messages in the network. On the first release of the tool the programmer can copy and paste a captured message in hexadecimal format and transformed into a decimal format that is presented in a table format. The table format allows a programmer to understand, verify and validate the contents of the frame.

 

This software tool was developed using C# and Visual Studio 2013 IDE, it can be used on all Windows operating systems 32 and 64 bit. Programmers at Lord and Company keep improving, innovating and developing internal software tools that will help achieve the company’s vision. Future development of the software tool will have additional features to cross reference a messages with other data pre-defined data, save analyzed data into files, print and some others as needed.

 

machuca1

Figure 1. Window with input box and result output table.

Optimizing Motorola MDLC Over IP

Chris Phelps
Systems Software Engineer, Lord and Company, Inc.

This technical article for Motorola RTU networks using Ethernet links covers the messaging architecture of MDLC over Ethernet. How the MDLC is implemented within the IP structure, how the retry settings affect the messaging and what IP protocol is being used.
In order to document and validate some concepts of MDLC over Ethernet test in a lab environment have been performed. Two ACE 3640 processors with ID’s 1000 and 2000 communicate over Ethernet port (ETH1). An application was loaded into both units to send a message from site 1000 to 2000 using in some cases a TxFrm (transmit frame) and later a SndFrm (send frame) message type. (STS software version 15.50). In order to capture the network traffic between the units a port mirroring device in conjunction with a computer with running Wireshark software network diagnostics software has been used.
Motorola uses UDP (User Datagram Protocol) as opposed to TCP (Transmission Control Protocol). The difference is TCP guarantees the recipient will receive the packets in order by numbering them. The recipient sends messages back to the sender saying it received the messages. When using UDP, packets are just sent to the recipient. The sender won’t wait to make sure the recipient received the packet — it will just continue sending the next packets.
Test 1. The port configuration parameters of both units have default values. When a TxFrm type message is used between sites the following capture is observed on Wireshark:

Fig 1. Wireshark capture between site 1000 and 2000 using TxFrm. Successful communications between sites.

The 3 messages sent back and forth between sites are known as a three way hand shake. It is a three-step method that requires both the client and server to exchange SYN and ACK (acknowledgment) packets before actual data communication begins. The first message (denoted above as No. 1) is the “SYN”, the second message (denoted above as No. 2) is the “ACK” and the third message (denoted above as No. 3) is the actual Data Packet sent from the STS Application. Notice that when both sites are communicating the time delays between each event is the milliseconds order (as expected for a 10/100 communication link)
Test 2. The port configuration parameters of both units have default values. In this test the communication link between sites is failed or disconnected. When site 1000 use a TxFrm to site 2000 the following capture is observed:

Fig 2. Wireshark capture between site 1000 and 2000 using TxFrm.  Bad communication between sites

Site 1000 did three attempts of the “SYN” message to the destination site (Site 2000) with a 10 second delay between each try. These results are correlated back to the Advanced Port Settings of the unit:

 Fig 3. Partial view of ETH1 Ethernet advance port configuration parameters for sites 1000 and 2000.

The Poll interval parameter is the time delay between retries (10 seconds in this example) and the Maximum number of poll parameter is the number of retries to send a message (3 in this example).

The importance of this parameters is that it allows a programmer to adjust the time delay between retries and the number of “MDLC” retries, without any additional programming at the application level.  Configuring this parameter accordingly reduces the possibility of overruns when a message is transmitted.  Message overrun is a term we use to describe when the application attempts a retry prior to the MDLC retry attempts expiring, this could cause the transmission buffer of one site to become full and the potentially start dropping messages and affect the performance of the CPU.

Finding the optimum balance between time and number retries and listening time (RTU is not sending messages) can be key to fixing communication issues and streamlining communications. The optimization procedure can be extensive and have multiple iterations before finding the correct balance; considerations to the network structure, speed, data reliability and the particular application will have to be considered to make proper adjustments.

Another important feature of the test was to demonstrate the use of network capture tools such as Wireshark to do diagnostics and help on the parameter settings configuration using reliable data.

On the next article a similar test will be performed using a transmission type SndFrm (Send Frame) between sites.  We will show the parameters effect on the sites that communicate using this type of frame.