Hello People of the Raspberry Pi,
I have a memory leak!
Background: I am writing code for a very basic SCADA (Supervisory Control And Data Acquisition) system. Long story short I have a network of LoRa nodes across a farm and a central Yun gateway (pushed info to an API: Thingspeak). This is the field level control and it is done by a combination of Arduino and Lilygo processors. It looks at variables such as water level, temperature, humidity, wind speed and direction etc. This end works well. At the supervisory level I have added a Raspberry Pi 3B+ and communicate to it via a simple serial link to the Arduino Mega Serial1. This works well and simply writes 16 variables to the Raspberry Pi.
NOTE: The Yun gateway writes to Thingspeak which is limited in free licensing (4 channels at 8 variables each = 32). This works well but will quickly cost a lot as we have several hundred variables to process (there is no issue with the Yun gateway). Hence the Raspberry Pi. I can ssh remotely via Terminal and run the SCADA application. It seems to work albeit slowly. So the Pi will be the core data collector and will be remotely accessed (the farm is approx 600km away and is attended to every three weeks or so...a long drive).
The Issue: The 3B+ Pi reads the variable well but I have noticed the memory increasing over time. The 3B+ Pi will only last a day or so before the memory is saturated and the Pi hangs. I have researched this over and over without solution. I feel I am doing something fundamentally wrong.
The code below is an extract from my application and appears to be the root cause of the system memory leak. The aim of this code is to collect all incoming serial data and pass onto global variables which will be used in several places throughout the application such as a main GUI screen as well as alarm lists (databases) and popup GUI's etc. It is setup as an infinite loop (which I suspect is part of the issue). The data on the serial port needs to be continuously read (forever). The 3B+ will run continuously until failure (hopefully a few years+).
I have tried various methods to correct for the memory leaks e.g. garbage collect gc.collect() , reference counting to check how many times a variable is assigned, deleting variable memory etc. All of these methods did not work so I have #commented them out of the program but left in place so the reader can see what I have tried. I have also left some memory analysis code but mostly #commented out. This is for the convenience of the reader (it you decide to test my code). The stuff I left in place was the minimum needed to measure memory leaks.
Here is my code:
The above code is Python3 and compiled under Thonny. I have tried the same code on two different Raspberry Pi 3B+ and also a Pi 4B+ with the same memory leak.
Can anyone advise what I am doing wrong with my code? The application works well however with this memory leak the project is ultimately a failure. Any help would be most appreciated.
Regards,
Brian
I have a memory leak!
Background: I am writing code for a very basic SCADA (Supervisory Control And Data Acquisition) system. Long story short I have a network of LoRa nodes across a farm and a central Yun gateway (pushed info to an API: Thingspeak). This is the field level control and it is done by a combination of Arduino and Lilygo processors. It looks at variables such as water level, temperature, humidity, wind speed and direction etc. This end works well. At the supervisory level I have added a Raspberry Pi 3B+ and communicate to it via a simple serial link to the Arduino Mega Serial1. This works well and simply writes 16 variables to the Raspberry Pi.
NOTE: The Yun gateway writes to Thingspeak which is limited in free licensing (4 channels at 8 variables each = 32). This works well but will quickly cost a lot as we have several hundred variables to process (there is no issue with the Yun gateway). Hence the Raspberry Pi. I can ssh remotely via Terminal and run the SCADA application. It seems to work albeit slowly. So the Pi will be the core data collector and will be remotely accessed (the farm is approx 600km away and is attended to every three weeks or so...a long drive).
The Issue: The 3B+ Pi reads the variable well but I have noticed the memory increasing over time. The 3B+ Pi will only last a day or so before the memory is saturated and the Pi hangs. I have researched this over and over without solution. I feel I am doing something fundamentally wrong.
The code below is an extract from my application and appears to be the root cause of the system memory leak. The aim of this code is to collect all incoming serial data and pass onto global variables which will be used in several places throughout the application such as a main GUI screen as well as alarm lists (databases) and popup GUI's etc. It is setup as an infinite loop (which I suspect is part of the issue). The data on the serial port needs to be continuously read (forever). The 3B+ will run continuously until failure (hopefully a few years+).
I have tried various methods to correct for the memory leaks e.g. garbage collect gc.collect() , reference counting to check how many times a variable is assigned, deleting variable memory etc. All of these methods did not work so I have #commented them out of the program but left in place so the reader can see what I have tried. I have also left some memory analysis code but mostly #commented out. This is for the convenience of the reader (it you decide to test my code). The stuff I left in place was the minimum needed to measure memory leaks.
Here is my code:
Code:
import time#import osimport serialimport psutil#import gc#import sysglobal serial_latchglobal serial_bufferglobal Temp1aglobal Temp2aglobal Temp3aglobal Temp4aglobal Temp5aglobal Temp6aglobal Temp7aglobal Temp8aglobal Temp9aglobal Temp10aglobal Temp11aglobal Temp12aglobal Temp13aglobal Temp14aglobal Temp15aglobal Temp16a ser = serial.Serial('/dev/ttyUSB0', 115200)time.sleep(2)def serial_comms(): while True: memory_usage = psutil.virtual_memory() #ref_count=sys.getrefcount(memory_usage)-1 #print("ref_count memory useage:",ref_count) #memory_total=memory_usage.total #memory_available=memory_usage.available memory_percent=memory_usage.percent memory_used=memory_usage.used #memory_free=memory_usage.free #memory_active=memory_usage.active #memory_inactive=memory_usage.inactive #memory_buffers=memory_usage.buffers #memory_cached=memory_usage.cached #memory_shared=memory_usage.shared #memory_slab=memory_usage.slab #print("memory_total",memory_total) #print("memory_available",memory_available) print("memory_percent**************************************** ",memory_percent) print("memory_used",memory_used) #print("memory_free",memory_free) #print("memory_active",memory_active) #print("memory_inactive",memory_inactive) #print("memory_buffers",memory_buffers) #print("memory_cached",memory_cached) #print("memory_shared",memory_shared) #print("memory_slab",memory_slab) print("Serial port reading") ser.flushInput() y = ser.readline() if (y == b'120\r\n'): # Retrieve the data from the serial port and convert it to float for i in range(16): temp = ser.readline().decode().strip() globals()[f'Temp{i+1}a'] = float(temp) print("Temp",i+1,"a",temp) time.sleep(0.1) #gc.collect() #ref_count=sys.getrefcount(Temp4a)-1 #print("ref_count :",ref_count) print("Temp4a",Temp4a) #del memory_usage, memory_total, memory_available, memory_percent, memory_used #del memory_usage, memory_total, memory_available, memory_percent, memory_used, memory_free, memory_inactive, memory_buffers, memory_cached, memory_shared, memory_slab time.sleep(0.5) #gc.collect() print_test()def print_test(): print("PRINT TEST of Temp4a",Temp4a)serial_comms() Can anyone advise what I am doing wrong with my code? The application works well however with this memory leak the project is ultimately a failure. Any help would be most appreciated.
Regards,
Brian
Statistics: Posted by Brian Usher — Sat Dec 30, 2023 12:56 am