Author |
Topic |
|
ullix
Germany
1171 Posts |
Posted - 01/02/2017 : 05:10:53
|
I programmed a Python script to access my new GMC-300Eplus using Linux Ubuntu Mate 16.04.
Guidance for programming is acknowledged to be taken from Phil Gillaspy's C++ code for Linux, also found in the GQ download page (http://www.gqelectronicsllc.com/downloads/download.asp?DownloadID=68 )
My code can be freely downloaded here https://sourceforge.net/projects/geigerlog/
In its simplest, stripped down version, this is a fully functional Python data logger:
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import serial # the communication with the serial port
import time # time formatting and more
def getCPM(ser): # get CPM from device
ser.write(b'<GETCPM>>')
rec = ser.read(2)
return ord(rec[0])<< 8 | ord(rec[1])
ser = serial.Serial('/dev/ttyUSB0', 115200) # open serial port
while True:
ts = time.strftime("%Y-%m-%d %H:%M:%S")
cpm = getCPM(ser)
print ts, ", CPM=", cpm # print timestamp and CPM
log = open("./log/cpm.log", "a") # logs the data
log.write(ts + ", " + str(cpm) + "\n")
log.close
time.sleep(10) # sleep for 10 seconds
ser.close() # close serial port
A slightly more demanding routine uses numpy for fast processing of large data files, and produces a graphic using matplotlib, like the one attached (gaps in data from programming developments breaks).
Files can easily be modifed to adapt to your needs.
Image Insert:
|
|
Reply #1
ZLM
1261 Posts |
Posted - 01/03/2017 : 21:50:29
|
Great work! Thank you for sharing. |
|
|
Reply #2
rimbus
8 Posts |
Posted - 02/12/2017 : 02:11:02
|
Thanks for posting the scripts.
I encountered several problems:
* simple.py: the log file must exist (an empty file is sufficient) * plot.py: log file name has to be modified (eg logfile = './log/cpm.log' instead of '/home/ullix/Geiger/log/cpm.log'
Main problem: plot.py does not work for me. Error message:
line 141, in plot_overview2 ... ValueError: x and y must have same first dimension
Any idea what went wrong?
|
|
|
Reply #3
ullix
Germany
1171 Posts |
Posted - 02/12/2017 : 05:21:41
|
so the simple program was indeed very simple, and hard-coded-paths are a pain in the ...
But the error is indeed more serious. So the (first) problem is this line ?
plt.plot(logtime[N/2:recmax - N/2], np.convolve(cpm, np.ones((N,))/N, mode='same')[N/2:recmax - N/2], color="yellow", linewidth=6, label ="") The command wants to: plot(x,y,options) and the error says that x and y are of different length. Try it out, put these commands before it:
print "x=", len(logtime[N/2:recmax - N/2])
print "y=", len(np.convolve(cpm, np.ones((N,))/N, mode='same')[N/2:recmax - N/2]) and run. The numbers are different by what? 1, N/2, N, a lot more?
Comment the lines 141 1nd 142 out (they do the smooth red/yellow lines), and see what happens. It is about deep into the numpy stuff; may not be easy to figure out. |
|
|
Reply #4
ullix
Germany
1171 Posts |
Posted - 02/12/2017 : 09:21:22
|
rimbus: I ran into that same problem myself today ;-) The reason is a limited number of data points and the smoothing algorithm, and perhaps a bug in the numpy code.
I calculate the moving average (the red/yellow line) by calculating a convolution using the numpy functions. I had chosen to average over N=300 data points (line 140). When your log file has fewer data points than you want to take the averge from, you obviously have an issue.
The numpy convolve allows up to N*2+1 datapoints, and I believe, this does not even make sense mathematically (hence I think it is a bug, but the math is complicated).
To fix, change the line 140 into a more reasonable number, e.g.:
N = min(int(len(logtime) / 2), 100) or limit N by some other means.
|
|
|
Reply #5
rimbus
8 Posts |
Posted - 02/12/2017 : 12:06:24
|
Indeed, the problem was the small number of entries in my logfile. With larger database the program presents the plot (and now I can start to understand the underlying principles).
You wrote another test program. I posted a question on this. Perhaps you have an answer too? Your test program downloads the history pytes. But which bytes are cps/cpm and which bytes are time? And how to I get the time out of the bytes? |
|
|
Reply #6
ullix
Germany
1171 Posts |
|
Reply #7
Distelzombie
Germany
202 Posts |
Posted - 02/14/2017 : 12:41:56
|
Im going to use this script in Windows with newest Python version. Is it ok if I paste in the changed script? Some commands are different in python3 Im writing a simple HowTo because python devs hate Windows apperently.
I ran into a problem. What kind of logfile is used in the plot Script? I guess its made by the other script, cause any other logfile type I get is unuseable. (the .csv give wrong number of columns error i.e.)
I got the geiger_basic.py to work atleast a little bit. Im stuck with this error now:
Traceback (most recent call last):
File "C:\Users\Chaosuser\Desktop\GeigerLog\geiger_basic.py", line 256, in <module>
cfg= getCFG(ser)
File "C:\Users\Chaosuser\Desktop\GeigerLog\geiger_basic.py", line 134, in getCFG
cfg.append(ord(rec[i]))
TypeError: ord() expected string of length 1, but int found Made by this function:
def getCFG(ser):
# Get configuration data
# send <GETCFG>> and read 256 bytes
# returns the configuration data. A total of 256 bytes will be returned.
#
# This routine converts the 256 character long string
# into a python list of 256 int values, and returns the list
ser.write(b'<GETCFG>>')
rec = ser.read(256)
cfg = []
for i in range(0,256):
cfg.append(ord(rec[i]))
return cfg This Error comes from every other function instead getVER(ser). I guess its some kind of data stream error. If i change the baudrate to 57600 even getVER(ser) does not work. Only 115200 works kind of. Whats wrong?
PS: Im not a linux guy. What does that mean? Its written on top as commentary: "using serial communication via /dev/gqgmc (linked to /dev/ttyUSB0" |
GMC-300E+ V4.20 with sbt-11a alpha tube
My statements are "stuff-a-hobbyist-says" and not in any way professional. |
Edited by - Distelzombie on 02/14/2017 14:27:35 |
|
|
Reply #8
ullix
Germany
1171 Posts |
Posted - 02/15/2017 : 02:49:44
|
If you are not very(!) experienced in Python, I'd discourage the use of my Python2 version program and run it in Python3. The error you are seeing is a consequence of some of the differences. I did explain some of the issues here in reply #3: http://www.gqelectronicsllc.com/forum/topic.asp?TOPIC_ID=4453 In short, byte sequences are handled as strings in P2 and as byte sequences in P3.
The "using serial communication via /dev/gqgmc (linked to /dev/ttyUSB0" thing is about the way Linux handles what is called "COM-Ports" under Windows. Instead of "/dev/anything" as in Linux, under Windows you would use "COMX" with X any number from 0(?),1,2,... to which your Geiger is connected. Like you can have different numbers at the COM ports, so one can have different numbers at the "dev/ttyUSBX" designation, which becomes relevant, when you have multiple serial devices connected. The linkage allows to have a consistent connection to, in this case, "/dev/gqgmc", irrespective of other devices.
The log file must have precisely(!) this format:2017-02-12 15:51:52, 23
2017-02-12 15:51:57, 23
2017-02-12 15:52:17, 18
2017-02-12 15:52:22, 22 This can be changed, of course. In the main of the plot program it is this line:
data = np.genfromtxt(logfile, delimiter=",", converters = {0: datestr2num}) # use numpy to read data into 2-dim array plus this function:def datestr2num(string_date):
""" convert a Date&Time string in the form YYYY-MM-DD HH:MM:SS
to a Unix timestamp"""
dt=time.mktime(datetime.datetime.strptime(string_date, "%Y-%m-%d %H:%M:%S").timetuple())
return dt
I could not make this work under Python3. Please, let me know when you solved it! |
|
|
Reply #9
Distelzombie
Germany
202 Posts |
Posted - 02/15/2017 : 10:42:47
|
Why did you use python 2 in the first place? Its like programming c++ with the version from 10 years ago. Look at all the changes! They're not done to make your work more complicated. :)
Oh ... quote: (But I had trouble with numpy in P3, so I stayed in P2)
OK. :)
quote: The "using serial communication via /dev/gqgmc (linked to /dev/ttyUSB0" thing is about the way Linux handles what is called "COM-Ports" under Windows.
Oh, ok. Yeah it was one of the first problems i ran into. quote: The log file must have precisely(!) this format: ... This can be changed, of course. In the main of the plot program it is this line:
Ah, so you're writing your own, basically. Why dont you use the .csv made by the data logger software? Is there no such software for linux?
quote: In short, byte sequences are handled as strings in P2 and as byte sequences in P3. ... I could not make this work under Python3. Please, let me know when you solved it!
Ah, good to know! Thanks :D At first glance i would say just use a byte2string function. There probably is one build into python. If not, theres definetly some on the internet in form of a wheel. I'll take a look at this. |
GMC-300E+ V4.20 with sbt-11a alpha tube
My statements are "stuff-a-hobbyist-says" and not in any way professional. |
|
|
Reply #10
Distelzombie
Germany
202 Posts |
Posted - 02/15/2017 : 11:10:01
|
(Im using this message and edit it as i go along.) This helps me think. Bold = Commentary or stuff Newest problem in always at the bottom
Using this function...
def decodeTOstr(byte):
ser=str(byte)
return ser Then use something like this in every function that works with the string: rec = decodeTOstr(rec) I get a string i can work with ....
No it works until i get to the Date function. I have to resolve this error now. return datetime.datetime(idt[0], idt[1], idt[2], idt[3],idt[4],idt[5])
ValueError: month must be in 1..12 The output of a print is: 2098 39 92 120 49 49 ... that doesnt seem to work right. :/ But the string is garbage... Using the b'byte' doesnt work either. ----
It does work using this: def decodeTOstr(byte):
#ser=str(b"byte")
ser = byte.decode('latin-1')
return ser Now i get geiger_plot.py problems...
Basically, the geiger_basic.py does not write the file that plot.py needs. If i create it myself basic.py does not write in it which upsets plot.py ever more. If I paste in the example that you provided in this thread, plot.py has a problem reading it. It says: "strptime() argument 1 must be str, not bytes". Looks like the same byte to str error like in basic.py, but in this case the data is already in str form. pfff --- resolved using yet another .decode Next problem: print ("\t%8d %s %6d" % (i, time.strftime("%Y-%m-%d %H:%M:%S", time.localtime((logtime))), cpm))
IndexError: index 4 is out of bounds for axis 0 with size 4
I dont know what index 4 is, but cpm variable is float and that gives errors later on in plotting, so i guess logtime is too... making it int does not change the error. For example: last five record gives a -1 too
Last 5 records:
-1 2017-02-12 15:52:22 22
0 2017-02-12 15:51:52 23
1 2017-02-12 15:51:57 23
2 2017-02-12 15:52:17 18
3 2017-02-12 15:52:22 22 Maybe that index 4? Hm omg, are you kidding me? I expects 5 records and gives error if there are less in the logfile. You should do some error-checking xD Current Problem: plt.plot(logtime[N/2:recmax - N/2], np.convolve(cpm, np.ones((N,))/N, mode='same')[N/2:recmax - N/2], color="yellow", linewidth=6, label ="")
TypeError: slice indices must be integers or None or have an __index__ method I guess it somehow doesnt like the conversion to unix time... -It does convert the date and time correctly to unixtime though. -It does work correctly using this unixtime. I mean it shows "last 5 records" for example. -hm... -converting cpm and logtime arrays into integers does not resolve the problem. -aaaaah! I didnt realize "[N//2:recmax - N//2]" includes mathematical operations. It needs two "/" as you see above.
Next problem: File "C:\Program Files\Python35\lib\site-packages\matplotlib\axes\_base.py", line 244, in _xy_from_xy
"have shapes {} and {}".format(x.shape, y.shape))
ValueError: x and y must have same first dimension, but have shapes (0,) and (5,) This seems easy to fix... No its not... Dont know what causes this. Still a bug. I deactivated the creation of an average line so that i can go on. Its basically works now. The plotting works and reading of the file is fine. But I dont get CPM from the device. So, currently two Bugs.
I get the cpm data now, but conversion into decimal is not working correctly... :/ see DEBUG getCPM//; rec= b'\x00\x17'
2017-02-16 00:16:42 CPM= 25127
DEBUG getCPM//; rec= b'\x00\x18'
2017-02-16 00:16:43 CPM= 25127
DEBUG getCPM//; rec= b'\x00\x19'
2017-02-16 00:16:44 CPM= 25127 (I included the Debug line to see whats happening) You see, you can get the CPM every second too.
IT WORKS FINALLY! ... well except the average line. Image Insert:
54194 bytes It is actually possible to read CPM every second from the device as you can see here:2017-02-16 01:08:51, 62549
2017-02-16 01:08:52, 63509
2017-02-16 01:08:53, 64423
2017-02-16 01:08:54, 65317
2017-02-16 01:08:55, 651
2017-02-16 01:08:56, 1534
2017-02-16 01:08:57, 2439
2017-02-16 01:08:58, 3312 (You can see the wrap-around here if it goes over 65535cpm) Some issues: -I cant reach the buttons under the plot for some reason. -Average line does not work -Unixtime is shown under the plot -I have to make something that works around the problem of the maximum 65535CPM. As you see in the picture, it cannot go over 65535, it falls down to 0 and counts up again. -Everything is hardcoded I wrote a rudimentary user interface which asks for every value except the colors. Can be mostly skipped by pressing 0. Also included a question if you want to plot the data right know or later, so that you can leave the program running in background without being interrupted every 20s... I can send you the files so that you can include them into youre sourceforge account. |
GMC-300E+ V4.20 with sbt-11a alpha tube
My statements are "stuff-a-hobbyist-says" and not in any way professional. |
Edited by - Distelzombie on 02/15/2017 19:00:05 |
|
|
Reply #11
ullix
Germany
1171 Posts |
Posted - 02/16/2017 : 02:56:52
|
The conversion between byte and string is simply done by ord() and chr(). Don't bother with character codes, it is not only irrelevant here but introduces errors to the data.
Before you touch the numpy stuff, better become very(!) familiar with Python.
When you get CPM readings near the 2^16 limit, you should switch device to CPS readings.
How on earth did you get a radiation source to yield those readings? |
|
|
Reply #12
Distelzombie
Germany
202 Posts |
Posted - 02/16/2017 : 07:15:23
|
Do you mean the yellow and orange average lines with "numpy-stuff"? That wasnt extremely hard.
quote: When you get CPM readings near the 2^16 limit, you should switch device to CPS readings.
Huh... i guess i cant use that in this program as long as im randomly switching to it inbetween. I'll ask the user at startup. The source is uranium ore. And i have a alpha sensivite tube installed.
These two average lines work now. Im having a problem with the x-Axis: The first count is plotted 'way too late'. In this case the first one is almost at the 20000 line, which makes no sense regarding the description. The 20000 is a weird number you have to add to the number in bottom right corner to get the unix time. This is useless and obviously not "seconds since first record". At the mark of 20000 unix time therefore was 1487220000 which is: 16.02.2017 - 05:40:00 and thats correct. (GMT+1) Im working on a replacement of x axis labels but it isnt easy to still have a perfect correlation to the cpm values.
The problem with the red average line was that you made N=300. But if there are less than 300 records in the file it doesnt work. I fixed this simply by using an IF that generates a value for N which is N=len(logtime)+1/2 if logtime is less than 300 logs big. This makes the line more wiggly in first minutes depending on your value of the logging frequency, but you're probably not using these plots anyway. omg ... i just read that you already fixed this problem at least in this forum. Why havent i read carefully?
I also want to start geiger_plot.py in a new shell so that you dont lose time in the record while its plotting the data. (2s minimum) But my computer doesnt forward the argument of the filepath to the script even if I start it from command line. Tried every fix already... hmpf OK that has been implemented, every Bug has been resolved and i wrote a user interface (I only had to ask two things on stackoverflow.com - a new record! xD) DO you want to include my files to your sourceforge, ullix? I didnt change ___author and stuff. Give me your email or some other means of transportation. Otherwise I will change it, give you credits, create my own sourceforge thing and share the links here. |
GMC-300E+ V4.20 with sbt-11a alpha tube
My statements are "stuff-a-hobbyist-says" and not in any way professional. |
Edited by - Distelzombie on 02/17/2017 13:15:39 |
|
|
|
Topic |
|
|
|