Carleton Weather

Five days remaining for me, and in the ten since my last writing the GUI has grown to maturity. It displays raw and converted data (and should grow automatically when new strings and fields are added); it displays text alerts (info and error messages); it has buttons for all sorts of commands to the PIC; it refreshes every 20 seconds; it remotely starts and stops the program on Freya; it displays the freshness of the data shown and the status of Freya and the PIC. This sort of many-featured, event-driven programming is proving to tend quickly toward complexity, so I’ve rewritten the code a number of times and am pretty happy with how it stands now. It’s a GUI, after all, so here’s a picture:

Along the way I’ve been adding new features liberally to the program on Freya, which now also maintains a few files containing information for the dash to display. The documentation is also coming into being — though I don’t know what use it is, as I gather Tom and Doug mean to have me continue working on Freya as the tower grows up. Still, I might forget in the meantime how to add new fields…

The status of the project at large: Freya is ready and waiting for deployment. Tom is working on instruments — right now we’re receiving data from about 2/3 of the bottom section (the top is not yet started), though not all of those 2/3 are sending real data, I think. Paul has a better version of the website debugged and is now doing mysterious database-side things. We (Paul and I) have cleared the eventual location of the tower of foliage, and Tom is figuring out the radio link that will connect the tower to the outside world. I’ve been doing some fine-tuning of the scheduling, and have brought the drift of Freya’s cycles down from +1.2 ms/cycle on average to 0 ms/cycle last time I measured, which may be important for the integrity of Freya’s communications with the GUI and PIC, but not very important.

The python program which will run on Freya is done, tested, and chugging away happily. Yesterday and today I have been working on a dashboard for Tom, of which the very first functional version is now, well, functional.

The dashboard is just a window with a refresh button and two tabs, one of which displays all received raw data and the other all converted data. It is coded in python using Tkinter (which really had me hating GUI programming for a while there, before I either got the hang of it or merely stumbled on things that were correct) and pysftp, a wonderful little module that  wraps paramiko and provides more-than-adequate and hands-off sftp functionality. I tweaked handleData.py to set aside data for the dash in a file the dash can easily grab and parse, and the whole program ended up under 200 lines. I’m treating it more as a proof-of-concept and intermediate tool than as finished code — it’s not pretty and hardly commented. Version 2.0 will be cleaner, more debuggable, and more transparent. It will automatically refresh data and will also display error messages.

Tom’s computer is on Windows 7, so rather than having him run it from the command line I turned the python program into a windows .exe file using py2exe. This entailed a bit of dependency-chasing and a lot of learning-how-to-use-windows-command-line, but it now runs nicely from a desktop shortcut, which from my point of view is victory.

Aside from updating the dash, I still need to write documentation for Freya and add new fields. Doug also may have some new responsibilities for the board for me to implement.

–Soren

Today I wrote and tested a DataOutputter class, which was the last piece of the program to be written (short of adding more converter objects, which can happen at leisure). The Outputter receives data one field at a time and then, when prompted, saves all the data to the flash drive and to 2UNIX. The automated backup system works like a charm — I can mount and unmount the flash drive at will and end up with all the data in order in one file on the USB, which is more fun than it probably sounds.

All the individual pieces are pretty well tested and debugged — what remains is to test them all together and make sure the system works as expected. I can spend all of tomorrow doing that, before moving on to writing documentation and the control panel next week.

–Soren

Today I wrote a script to keep the board’s time accurate, which works as follows:

1. Uses service to pause ntpd to free up the NTP socket.
2. Runs ntpdate with servers ntp.carleton.edu and time.nist.gov.
3. Stores the exit code of ntpdate. If it failed, get system time from the hardware clock. If it succeeded, sync the hardware clock to new system time.
4. Send an alert if unable to access hardware clock.
5. Restart ntpd and exit.

This will be run hourly by cron at low priority, and by the script starting the program. This way, the hardware clock is always as correct as possible in case it must be used. The use of ntpd may cause a large discontinuity in system time, but these will be forced before the program begins to operate — the sync to hardware time will reduce this discontinuity to a easily manageable level unless the board has been running without internet for a week or longer, in which case (approximately) just one cycle’s worth of data per week of internet downtime will be disrupted when internet is restored. This would manifest simply as missing data from that cycle; the scheduler would rush the wait time between sending a Q to the PIC and requesting data with an S, and find nothing when it tried to read.

The primary benefit of this way of keeping time is that received data is guaranteed to be timestamped accurately to within a few seconds at the very most. Failure cases are massive internet downtime and RTC battery failure, both of which are reported to users. If the RTC battery dies and power/internet go down, and then power is restored without internet, there is no avoiding that timestamps will be off by however long power was down for. A feature to add if I finish everything else is smart tracking of this scenario — the python program knows if the RTC is accessible, and throws out data collected with no RTC or internet access.

Before that, however, I still have to:

1. Test scheduling and startup.sh more fully.
2. Write and test a DataOutputter class
3. Write some more data converter objects as available
4. Write Tom’s control panel
5. Write documentation

–Soren

One week since my last writing, and the code-scape is looking already very different. I decided today that this was one of those projects requiring at least one rewrite from half-scratch, so  I’ve been refactoring in an effort to fight the spaghettitude that was already making things difficult. The new and even more object-oriented organization is:

• raw data is held in a container which
  • (a) does all format-checking of received strings, and
  • (b) can provide raw data requested by label

This is requiring some comparatively heavy code, but it is covered up by the clean and flexible interface methods get(label) and store(). The major benefit is that this way, string format can change and the code to access fields won’t — if one method needs geiger_ticks, it can always get geiger_ticks by calling container.get(“geiger_ticks”). This class is informed by another new thing…

• format is given by two text documents, one containing expected format of received data and the other containing the format in which to output data.

Short parsers turn these files into dictionaries holding the proper locations of incoming and outgoing data. These dictionaries live in the containers, speaking of which:

• converted data is held in a container which
  • (a) formats the data it is given piece by piece, and
  • (b) saves the data in the appropriate files when told to

Data is given to this container by…

• datafield objects have been updated to map to output fields rather than input fields, which is more robust and flexible.
  • each datafield object will inherit from some DataField class that knows how to check range, keep track of error state, and so on

This is much like the previous system except it does not assume that each input takes only one output.

It is my hope that separating data handling from scheduling will make for better code and easier maintenance by Tom and Doug, who (this way) will only need to change a configuration file to alter formatting. Another part of the refactoring is that error notifications are now separated from the weather collector object. They live in errors.py and can be used easily by all pieces of the code.

As for something completely different: I have begun working on Tom’s control panel (using python and Tkinter). Simplifications necessary for that (ie a common formatting conf doc) prompted the changes to the main program I’m working on now. Time permitting, these should let me do some useful things — like automatically display new datafields in the control panel when they are added to the board’s conf docs.

• To Do (the list has grown):
  • finish writing InDataContainer (est. 2 hours)
  • write OutDataContainer (4 hours)
  • rework DataField class definition (1 hour)
  • move error things into errors.py (.5 hour)
  • retouch WeatherCollecter class definition to fit with these changes (1 hour)
  • start defining what datafields I have (1 hour)
  • finish control panel (7 hours?)
    • generate display from conf doc (3 hours)
    • send commands to board (2 hours)
    • get data from board(2 hours)

–Soren

Things are continuing apace. The startup script is written (in bash), and is responsible for mounting the USB, setting system time, running and renicing Core.py, and so on. This necessitated some monkeying around with sudo configurations and fstab, but now works smoothly. A similar graceful-end bash script uses SIGUSR1 (an idea from my 2010 predecessors) to tell Core to stop at its leisure, which right now means at the end of a cycle. These things are tested and work nicely. A particularly handy feature is that startup.sh catches error messages from Core should it fail unexpectedly and sends them on along with a notification that Core has crashed.

Data storage and manipulation is mostly done, pending some tweaking. That function proved to be a great deal of complex error-handling for things that are very unlikely to fail anyway, but I figured due diligence now might prove worthwhile if it saves a batch of data somewhere down the line. Meanwhile, definitions of how to convert raw data have already grown to 1700 lines even though I don’t know how to convert any of the data yet — most of those 1700 lines are generic or placeholders, generated rather quickly with the magic of Sublime Text 2 and to be filled in at a later date.

A daily system health and space check is underway, though I’m still thinking of clean ways to trigger it daily. Unresponsive PIC and received-string format errors are also implemented. I’m thinking about doing away with the toggle approach to clearing 5Minute2Unix currently in use in favor of just letting the mac clear the file whenever it reads, if at all possible.

Still to be done:

• Integrate daily laundry-list into normal running of the program
• Write some sort of documentation
• Add data-processing and response logic
  • Add framework for range-checking
  • Possibly rework the way formats are stored
  • Possibly rework the way Core loops through and processes data
• Create a control panel for Tom

This last could be done two ways: using python and Tkinter, or as a web app using javascript. I’m leaning towards the latter, which seems more user-friendly. The learning (or rather, remembering) curve would be a bit steeper for me, but browser access would be worth it and the refresher would be helpful. More info to come when it exists.

–Soren

As of today, most of the code has been written and tested on Freya with a mostly functional big PIC! Error handling and notification was first, and so far mostly involves try-except statements everywhere. There are three levels of email alerts and logging (debug, info, error) which can be configured easily. The actual sending of emails works like a charm; each cycle’s worth of error messages are saved and sent as a digest. This system is satisfactorily robust to power loss and internet loss. Commands work in a similar way: they are queued for a cycle and then sent all at once. This logic is written and partially tested.

Setup code is also written: it initializes and configures logging, connects to the serial port, and initializes various files. Eventually it ought also to do things like confirm correct time, reset toggles, and so on. More excitingly, the main scheduled loop is written and running successfully. Over the course of 20 seconds, it calls functions that do the relevant things. The schedule right now is:

• Write a Q to the PIC, wait 2s.
• Send pending emails, wait 8s.
• Write an S to the PIC, wait 1s.
• Read/process/save data, wait 5s.
• Send all pending commands, wait 4s.

Other things now written but in need of more testing: logic to format 2Unix data (I ended up using string.Template), logic to read data, and logic to pass data to the objects that will handle it. The code right now also wants organization and comments, which I’ll get to tomorrow.

On the hardware side, the next challenge may be mounting the backup USB. pmount is uncooperative on Vor, and regular old mount require root privileges. I’ll have to run some tests (ie on boot do previously mounted devices stay mounted?) to see if anything more than a manual mount is necessary. The current board requires manual mount, and there’s probably not much the program could do about some failure causing the drive to unmount…

The list of features to be added:

• Daily actions, including:
  • resetting error flags
  • checking space and RAM usage
• Configuring toggle files for 2Unix and end script
• Configuring saving of data
• Ending at the appropriate point in a cycle if remotely turned off
• Startup bash script
• Some unified way of range-checking and responding to data (to be implemented in ConvertData.py)

— Soren

I spent the last few days beginning to realize the plan of writing data fields as objects. There are several reasons for this change:

1. Each field needs to have some inkling of what state it’s in so that (eg.) it does not send Tom and Doug the same error message every 20s.
2. Each field needs to know a number of things about how it behaves: how to convert data, what to do with the converted data, how to respond to the data, and so on.
3. There is some logic that each field will share, some that most fields will share, and some that none will share. The cleanest way I can think of to do this is to have each field object inherit from a DataField class (which is now mostly written) that knows how to register errors, issue commands, and so on.

It would, of course, be possible to do this without a single object — but it would be messy and difficult to add new fields. Some details are still to be worked out (ie., how should data be checked against allowable ranges) that will become clear once I know how more about the behavior of individual data fields.

Similarly, for scope and modularity reasons Core.py is going to become object oriented as well. Like the last group did it, there will be a WeatherCollecter object handling the data collection processes. The two challenges that promise to require most of my attention are:

1. Organization! Core.py is going to have a lot of functions and complex logical structure, which is unlikely to jive well with the BASIC-like vision Tom has for the code. I’ll probably need to wrangle it into something sequential-ish while preserving the more pythonic bits (ifmain clause, connections to other modules, exception-based control flow, object-oriented style) simplicity has driven me toward.
2. Error handling! I haven’t yet written a line of code I don’t worry about needing to wrap in a try statement or other such error-handling thing. This wants balancing with elegance and cleanliness, as spaghetti-code is a source of errors in and of itself…

In bigger news: the RTC saga appears to have been resolved. After approximately 15 hours of searching, it turned out that the only thing wrong with the tool I was hoping to use (ts4200ctl) was the usage statement. The –getrtc option is in fact not to display the hardware time, as the usage says, but to set system time from hardware time. Many thanks to Michael from TS for clearing this up. The new plan for fail-safing time is to call ts4200ctl –getrtc as part of the setup of Core.py and then sync the RTC to system time every so often (maybe in the same daily job that resets error flags) so that when internet or power is next lost, the RTC is as accurate as possible. This method doesn’t even require knowing if ntpd is active and connected to a ntp server, since there’s no harm in syncing the RTC to a system time that was last synced with the RTC itself.

Now, this is more or less what Unix systems are intended to do anyway — but in my testing I found it could take up to 7 min for this to happen on Vor (and possibly that was ntpd in action, not the RTC) so it might as well be coded right into Core.py.

Other updates: email alerts are working (using python’s smtplib) and most code is done, except for that which requires (a) knowledge of how to convert and respond to data and (b) knowledge of how the PIC will send data. Both of these are hard to say at this point, so I expect they won’t be written until most of the infrastructure upstream is finished. The rest will probably be finished and tested by a week from today — fingers crossed. This includes:

• configuring save locations and backups
• error proofing
• including time stuff as discussed above
• writing code to send commands to PIC
• writing code to set up scheduler

–Soren

As Doug warned yesterday, it turns out there is a possibility of the board “chopping” the PIC’s data — timing out mid-read and not getting the whole string. Thus far, I have observed this only with a timeout of 0 seconds, but logic dictates that it is at least possible, though very unlikely, with nonzero timeouts. The greater the timeout or shorter the string, the lesser chance this has of happening, but unless the interval of writing and reading are exactly the same, by Murphy’s Law Freya will eventually chop the data and bork things up.

So, two options are to use a None timeout or read one character at a time. However, Tom is right now figuring out how the PIC will make data available, and this point is likely to become moot, so we’ll put it on hold. What Tom was talking about was (since the PIC will likely have enough memory not to need to break data into many groups) sending two strings as queried for by the board, 3 times per minute. Two strings is a little messier than 1, but certainly workable.

This sort of structure will require more careful timing, so I’ve been looking at python’s sched library. I think it should be possible to run the whole main loop with only one scheduler object per go. The scheduler would be loaded along the lines of:

send Q

in 10s, send B

in 11s, read and process B data, and send T

in 13s, read and process T data

in 15s, save data

in 16s, send all control signals accumulated over cycle so far

in 17s, send all pending emails

in 20s, break loop and reload scheduler

where B and T are for bottom and top PIC groups. Details will be fleshed out when the PIC’s needs are clearer.

I’m thinking a nice clean formatting trick to format converted data on the fly will be to use .format() on a string that specifies the desired format to send to the database. Each data-converting function will process raw data, execute response/error logic, and then format its contribution into the big string. The string can be saved to oneline when done, and this doesn’t require order received to be the same as order outputted for maximum formatting ease. string.Template and substitute() seem to be the easiest implementation…

Another thing that’s happening is that Tom is beginning to draw up a spreadsheet with all data fields, their bounds, their conversion formulas, information, etc. From this, I can continue putting together ConvertData.py. And once I know the PIC’s timing schedule, I can write Core.py and BlockFormat.py. What I can work on now is how I’ll write error-handling and logging, which will probably be the most complex part of this whole operation. Necessary features will include:

• info/error/warning logging into a file on the board
• debug logging that can be turned on or off
• automatic responses to certain engineering data
• emails sent every so often containing alerts

Logging can be handled using the python logging library. Auto-response logic should be case by case, but easy. Control signals generated can be held in a file until time to send them over, and same for emails. What’s not clear is how I can cleanly avoid sending duplicate emails every 15 seconds. Maybe the planned GUI-type control and engineering application will grab those signals after the first email? Maybe I should make the data-fields objects so they know their notification history. In any case, I’ll be back next Monday and know more then.

–Soren

We got our serial port working today, thanks to a mini weather station Tom rigged up. After some brief debugging we were happily sending commands and receiving data. I’ve since tested a number of cases: receiving manually requested data, reading unrequested data on a loop, requesting an reading data on a loop. All seems to work well, and we shouldn’t run into any problems with lost data as long as the interval between read checks is less than the interval between writes from the PIC. The magic numbers: port=2, and baudrate=57600.

The next question, at least in my mind, is how Freya and the big PIC are going to talk. The first part of this is:

Do we want the PIC to send one message per data field or one message per group of sensors?

• Advantages of one per data field:
  • Each message could be cleanly keyed by a numeric ID corresponding to the data field, so parsing messages would be far simpler and cleaner on the TS 4200 side
  • Dealing with once-per-ten-seconds data fields would be easily solved by having all 6 have distinct IDs
  • Each data field could have just one identifier which could also be used to specify its place in the database
• Advantages of one per group:
  • The timing would be much more forgiving because there would be fewer strings/minute
  • Less demand on big PIC, more on group PICs

On the balance of things, I’m thinking one message per group would be simpler probably on both sides of the port. In the configuration file, there’ll have to be a two dimensional array keyed by (group, number) referencing a callable, and perhaps another referencing its location on the final datastring. The callables for each field should be defined in their own file. My ideal would be that the main program will not have to be altered when a sensor is added; an entry would be added to the table(s) in the config file and a function containing how to process its raw data would be defined in the data processing file. Constraints: sensors would have to be added to the end of their group numerically, but that’s the simplest way to do it anyway. For tomorrow: start writing the core of the program.

I think that the main program will be quite simple. Almost all error handling can be dealt with in the data processing file, and auto-control logic can be dealt with in its own file.

Another thing to be thinking about is how to create a UI of some sort for Tom to use to remotely talk to instruments — shouldn’t have two access points to freya running at once, so maybe we’ll have to route that through the Mac.

–Soren