Write data to file in Python?

[moeburn]

I'm using my Android smartphone to send accelerometer sensor data wirelessly to my PC via UDP, and I have a Python script to display the data on screen:

import socket, traceback

host = ''
port = 5555

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

while 1:
 try:
  message, address = s.recvfrom(8192)
  print (message)
 except (KeyboardInterrupt, SystemExit):
  raise
 except:
  traceback.print_exc()

And it works fine, but I don't really know anything about Python.  I want to write the data to a CSV file instead of the screen, how would I do that?

[Lant]

Adapted from http://docs.python.org/2/library/csv.html#csv.writer

import csv
with open('eggs.csv', 'wb') as csvfile:
    spamwriter = csv.writer(csvfile)
    spamwriter.writerow([message])

Also if you are using python 2.x you might want to add this to the top of your script (see: http://docs.python.org/2/library/__future__.html)

from __future__ import print_function, with_statement

[moeburn]

I'm using Python 3.3.2, I just installed it.  I haven't really programmed anything since the days of Visual Basic 6.

 

So I wasn't sure where to add that code you mentioned, so I added it like such:

import socket, traceback
import csv

host = ''
port = 5555

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

while 1:
 try:
  message, address = s.recvfrom(8192)
  print (message)
  with open('eggs.csv', 'wb') as csvfile:
    spamwriter = csv.writer(csvfile)
    spamwriter.writerow([message])
 except (KeyboardInterrupt, SystemExit):
  raise
 except:
  traceback.print_exc()

And I get these errors:

 

[Karl L.]

You have the right idea, but it's probably not good to open and close the CSV file so many times. The following script runs with no exceptions on my Debian 7.1 box with Python 3.2. I assumed the type of data you would be sending and tested it with netcat, so it should work.

#!/usr/bin/env python3

import socket
import traceback
import csv

host = ''
port = 5555
csvf = 'accelerometer.csv'

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

with open(csvf, 'w', newline='') as csv_handle:
    csv_writer = csv.writer(csv_handle, delimiter=',')
    while 1:
        try:
            message, address = s.recvfrom(8192)
            print(message)
            csv_writer.writerow(message)
        except(KeyboardInterrupt, SystemExit):
            raise
        except:
            traceback.print_exc()

[moeburn]

Well, at least the CSV was written without errors.  Now it's just writing in the wrong format.  I've attached a screenshot with the proper data display format on the top left, the code on the top right, and the CSV result on the bottom.  I think those numbers are the ASCII codes for the actual text?  I tried adding , encoding='utf8' after newline='' but that didn't fix it.

 

[Karl L.]

The CSV module was writing the integer value of each byte of the string to the file. Unlike Python 2, Python 3 differentiates between a text string and byte string. The message being returned from the socket is a byte string. I probably should have checked the sanity of my output file last time. Try the following script instead:

#!/usr/bin/env python3

import socket
import traceback
import csv
import re

host = ''
port = 5555
csvf = 'accelerometer.csv'

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

with open(csvf, 'a', newline='') as csv_handle:
    csv_writer = csv.writer(csv_handle, delimiter=',')
    while 1:
        try:
            message, address = s.recvfrom(8192)
            print(message)
            
            message = str(message)
            if message[0] == 'b':
                message = message[2:-1]
            data = []
            for x in iter(message.split(', ')):
                if re.search('^[-]{0,1}[0-9]+\.[0-9]+$', x) != None:
                    data.append(float(x))
                elif re.search('^[-]{0,1}[0-9]$', x) != None:
                    data.append(int(x))
                elif re.search('^[-]{0,1}[0-9]+\.[0-9]+,[-]{0,1}[0-9]+\.[0-9]+$', x) != None:
                    real, img = x.split(',')
                    real = float(real)
                    img = float(img)
                    data.append(complex(real, img))
                else:
                    print('unknown', x)
            csv_writer.writerow(data)
            csv_handle.flush()
        except(KeyboardInterrupt, SystemExit):
            raise
        except:
            traceback.print_exc()

If anyone else would like to test it or attempt further modifications, I recorded the values in moeburn's last screenshot in a text file and wrote a simple script to read the file and send its contents to the accelerometer script.

 

Test data file:

7406.60063, 3, 0.765, 1.334, 9.601
7406.70039, 3, 0.765, 1.374, 9.601
7406.77045, 3, 0.765, 1.334, 9.601, 5, 28.000, 8.125,-34.938
7406.80038, 3, 0.765, 1.294, 9.601
7406.11339, 3, 0.804, 1.294, 9.601
7407.17048, 3, 0.745, 1.334, 9.601
7407.31351, 3, 0.785, 1.324, 9.601, 5, 29.062, 8.000,-34.562
7407.37044, 3, 0.785, 1.383, 9.601, 5, 28.312, 7.750,-34.438
7407.41334, 3, 0.785, 1.324, 9.601
7407.44040, 3, 0.824, 1.324, 9.601
7407.53046, 3, 0.824, 1.363, 9.601, 5, 28.125, 7.375,-34.062
7407.60038, 3, 0.824, 1.363, 8.630
7407.61541, 3, 0.765, 1.520, 9.571, 5, 28.562, 7.250,-34.938
7407.62152, 3, 0.765, 1.471, 9.512
7407.63199, 3, 0.765, 1.304, 9.512
7407.65909, 3, 0.765, 1.373, 9.718
7407.66674, 3, 0.765, 1.373, 9.758
7407.68064, 3, 0.706, 1.373, 9.915
7407.70041, 3, 0.706, 1.374, 9.856
7407.72062, 3, 0.706, 1.324, 9.924
7407.73449, 3, 0.706, 1.324, 9.689
7407.74213, 3, 0.902, 1.265, 8.120
7407.75961, 3, 0.853, 1.432, 9.022, 28.750, 7.188,-34.125

Test script:

#!/usr/bin/env python3

import socket

host = ''
port = 5555
test = 'data.txt'

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.connect((host, port))

with open(test, 'r') as test_handle:
    for line in test_handle:
        line = line.rstrip()
        s.send(line.encode())

[moeburn]

Thanks for doing all that for me!  Sorry, I should have posted the sample data as text not a screenshot, but I don't know how to copy text from the Python output screen in Windows.  

 

Your latest example does a lot of processing to the data, even the data on screen is now coming up different.  But there's a lot of values missing from the CSV, here's a snipped form the CSV:

15396.84775,3
15396.85702,3
15396.86359,3
15396.87,3
15396.87998,3
15396.89283,3
15396.90079,3
15396.91003,3
15396.92027,3
15396.93003,3
15396.94188,3
15396.95006,3
15396.9702,3,5
15396.98038,3
15396.9963,3

As for the data being sent, here's a breakdown:

timestamp, accelSensorID, accelX, accelY, accelZ
7407.60038, 3, 0.824, 1.363, 8.630

occasionally magnet sensor also sends data when magnet sensor input changes:

timestamp, accelSensorID, accelX, accelY, accelZ, magnetSensorID, magnetX, magnetY, magnetZ
7407.61541, 3, 0.765, 1.520, 9.571, 5, 28.562, 7.250,-34.938

[moeburn]

Ok I got a much simpler code that writes to the CSV like this:

1,5,8,4,9,.,8,1,1,6,3,",", ,3,",", , , ,0,.,0,0,0,",", , ,5,.,0,9,9,",", , ,8,.,9,2,4
1,5,8,4,9,.,8,2,9,8,8,",", ,3,",", , , ,0,.,0,0,0,",", , ,5,.,0,5,0,",", , ,8,.,8,6,5
1,5,8,4,9,.,8,4,9,7,5,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,7,5,6,",", , ,8,.,6,4,9,",", ,5,",", , ,2,2,.,6,8,8,",",-,1,1,.,6,2,5,",",-,4,5,.,1,2,5
1,5,8,4,9,.,8,6,2,8,9,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,6,8,8,",", , ,8,.,6,4,9
1,5,8,4,9,.,8,7,2,0,2,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,6,8,8,",", , ,8,.,7,0,8
1,5,8,4,9,.,8,8,2,4,0,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,6,8,8,",", , ,8,.,7,7,7
1,5,8,4,9,.,8,9,3,5,3,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,7,4,6,",", , ,8,.,8,2,6
1,5,8,4,9,.,9,1,3,4,8,",", ,3,",", , , ,0,.,0,4,9,",", , ,4,.,7,0,7,",", , ,8,.,8,2,6

But I don't want all those comma delimiters in between each byte, because the android app actually adds the commas itself.  Here's the code:

#include libraries n stuff
import socket
import traceback
import csv

#assign variables n stuff
host = ''
port = 5555
csvf = 'accelerometer.csv'

#do UDP stuff
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

#do CSV stuff
with open(csvf, 'w', newline='', encoding='ascii') as csv_handle:
    csv_writer = csv.writer(csv_handle)
    while 1:
        try:
            message, address = s.recvfrom(8192) #get data from UDP
            print(message)                      #display data on screen
            csv_writer.writerow(message.decode(encoding='ascii'))      #write data to CSV
        except(KeyboardInterrupt, SystemExit):  #close on exit
            raise
        except:
            traceback.print_exc()               #display errors on screen

all I did was change csv_writer.writerow(message) to csv_writer.writerow(message.decode(encoding='ascii')), and removed the delimiter=',' , although that didn't actually remove the delimiter for some reason.

[astropheed Veteran]

I was wondering why all the RegEx when a simple x.decode() would suffice.

 

Can you do me a favor and change that 'while 1:' to 'while True:', it irks me lol.

[moeburn]

Ok I got it working now!  Apparently all I needed were a couple of [] brackets around the message.decode.  Here's the final code:

#include libraries n stuff
import socket
import traceback
import csv

#assign variables n stuff
host = ''
port = 5555
csvf = 'accelerometer.csv'

#do UDP stuff
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

#do CSV stuff
with open(csvf, 'w', newline='', encoding='ascii') as csv_handle:
    csv_writer = csv.writer(csv_handle, delimiter=',')
    while 1:
        try:
            message, address = s.recvfrom(8192) #get data from UDP
            print(message)                      #display data on screen
            csv_writer.writerow([message.decode(encoding='ascii')])      #write data to CSV
        except(KeyboardInterrupt, SystemExit):  #close on exit
            raise
        except:
            traceback.print_exc()               #display errors on screen

[moeburn]

If anyone is interested, here is an example of some data I recorded using this script:

 

 

Because the accelerometer is so sensitive, I intend to use it to measure my heart rate and breathing patterns (already confirmed possible) by taping the phone to my chest while I sleep.  That's why I needed the live stream of data to my PC; in-phone sensor recording apps weren't reliable enough after a few minutes of recording.

[moeburn]

Hey would anyone feel like helping me improve it a bit more?  At the moment, it only writes to the file once the program is closed.  I'd like it to be a little more reliable than that, in case of a power failure or a Windows crash.  I only know how to code it to write every time it receives a data sample, but that's 60 times a second, that's probably too often.  Would anyone like to write me a piece of code to only write to the CSV once every 100kB or so?

[Eric Veteran]

If it doesn't have to be CSV you could always try using pickle/cPickle, but you have to be careful with it as it can be a security risk if your application tries to unpickle data that has been intentionally malformed.

 

cPickle is quite fast and is the default over pickle in Python 3+.

[Karl L.]

Hey would anyone feel like helping me improve it a bit more?  At the moment, it only writes to the file once the program is closed.  I'd like it to be a little more reliable than that, in case of a power failure or a Windows crash.  I only know how to code it to write every time it receives a data sample, but that's 60 times a second, that's probably too often.  Would anyone like to write me a piece of code to only write to the CSV once every 100kB or so?

 

According to the documentation for the Python open() function, you can use the buffering argument to control the size of the buffer, and thus how often data is flushed to the CSV file on disk.

#!/usr/bin/env python3

import socket
import traceback
import csv

host = ''
port = 5555
csvf = 'accelerometer.csv'
buff = 100 # Flush the accelerometer data to disk once every buff KB

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

with open(csvf, 'w', buff * 1024, encoding='ascii', newline='') as csv_handle:
    csv_writer = csv.writer(csv_handle, delimiter=',')
    while True:
        try:
            message, address = s.recvfrom(8192)
            print(message)
            csv_writer.writerow([message.decode(encoding='ascii')])
        except(KeyboardInterrupt, SystemExit):
            raise
        except:
            traceback.print_exc()