<read>blog.stavi.sh</read>

Bicycle Commuters Overtake Train Commuters In The District Of Columbia!

Synchronous microcontroller communication interfaces. Everything you wanted to know, but were afraid to ask.

Synchronous Interfaces (suitable for peripherals on the same board or <1 m):
I2C (Inter Integrated Circuit, I squared C)
SPI (Serial Peripheral Interface Bus )
U(S)ART (Universal Synchronous & Asynchronous Receiver Transmitter)
(aka Serial Communication Interface (SCI))
(S) can be synchronous

 I2C
* true bus, requires only two pins for multiple devices (address space)
* 100kbits/sec, 400kbits/sec, 1Mbits/sec
* half-duplex
* high noise sensitivity, lower data integrity
* slave ack / rx confirmation

SPI
* not a true bus, N devices require 3+N pins
* 10Mbits/sec
* full–duplex capability
* any message sizes, suited for longer data streams (I2C limited to 8 bit word)

USART
* standard serial connection (also over USB)
* few hundred bits per second (bps) up to 1.5Mbps
* error checking with parity bit

Unit Testing Arduino With Python

Uses python to dynamically load Arduino hex images for unit testing. Supports windows and osx via configuration file. To find out where your hex images are placed by the Arduino IDE, hit the shift key before you hit the build (play) button. You can hit the shift for upload as well to find out where your avrdude (command line upload utility) is located.  Alternatively, you can look at the included config files and use your install location (using Arduino 0020).

http://github.com/toddstavish/Python-Arduino-Unit-Testing

Algerian Cities Spreadsheet, Latitude \ Longitude, And Map

Spreadsheet:
http://www.stavi.sh/downloads/algerian-cities-lat-long.xls

Google Docs:
https://spreadsheets0.google.com/ccc?key=tGdADtMhFEl1a10ruijx2Kw&hl=en

Socrata:

Afghani Cities Spreadsheet, Map, Latitude And Longitude

Spreadsheet:
http://www.stavi.sh/downloads/Cities_Within_Afghanistan_Latitude_Longitude.xls

Google Docs:
https://spreadsheets.google.com/ccc?key=0At92oU3FPZ4QdHhfMFJ0a2lUcXNuN0J2bHRR...

Socrata:

Real-time Relationship Analytics From Large-scale Graph Processing

Cassandra excels at storing large, active, decentralized datasets. Additionally, Cassandra’s rich data model allows efficient use for many applications beyond simple associative arrays. One interesting application is the processing of large-scale graph structures.

I have devised a graph application layer to extract and process social network analysis data from Cassandra, using InfiniteGraph. The technical benefits of the social-graph-extract application layer and its use of graph-oriented processing have been articulated.

Social network analysis is one application of a more general category, relationship analytics, as defined by Curt Monash. The relationship analytics problem domain maps well to the unique features of the Cassandra-InfiniteGraph hybrid system:

• dedicated vertex/edge API
• data can be clustered according to vertex/edge proximity
• disk-based/memory-centric access
• peer-to-peer communication from InfiniteGraph node to Cassandra node
• bidirectional updates between raw Cassandra data and Infinitegraph analytics
• parallel streaming and caching from InfintiteGraph
• modeling flexibility to support a variety of sources
• redundancy and high-availability
• precision and speed for graph analytics
• finding extremely long paths, all paths, unknown paths, or paths of nontrivial or indeterminate length

Current business problems that can utilize these features:

• analyzing high-frequency trading
• discovering high degrees of mutual interconnection in social networks
• data mining subtle retail correlations
• product recommendation engines
• determining terrorist or criminal behavior inferred from known relationships
• finding a pattern of relationships for fraud detection
• investigating the directed relationships between proteins and genes
• checking which entity has the shortest average connection to a group of others for cyber security (botnet controller)

The working codebase for this Cassandra / InfiniteGraph integration can be retrieved from GitHub. Forking of the main project is welcome (including downstream updates). If you have any questions or suggestions, please contact @toddstavish.

Python code that writes a multibyte value (long/float) over serial to Arduino

Visualizing Public Policy Polling Survey Data Of Delaware Voters (August 7-8, 2010)

The Daily Kos posted commentary and analysis of a recent Public Policy Polling survey of Delaware voters. The raw data was also posted. I loaded the raw data into Socrata’s social data platform to do some quick visualizations and corroborate the analysis.

After loading the raw data CSV, the first thing that I did was add meta-data into the column labels in Socrata. To do this, I cut and pasted the text descriptions of the questions from the survey PDF to the original raw data column label. You can see the meta information when you hover-over the columns in the tabular view or in the dataset summary; which also includes usage statistics, attribution to PPP, and the orignal PDF and CSV (embedded as attachments).

The benefit of adding meta-data is that anyone that looks at the raw data can now understand the values of the columns quickly. Additionally, in Socrata, the meta-information is decoupled from the actual data, so the data values can change without losing the semantic labeling. Generally, the meta information and data can be combined as part of the ingest process, the manual step was only required for this data-set because of the nature of the existing information. Note, my preferred choice for ingesting is via automated workflow using the web services API.

Markos’ commentary focused on questions 8 and 9 respectively:

• Q8: If the candidates for US Senate this fall were Republican Mike Castle and Democrat Chris Coons, who would you vote for?
• Q9: If the candidates for US Senate this fall were Republican Christine O’Donnell and Democrat Chris Coons, who would you vote for?

To verify the voting result percentages from the survey raw data, I grouped the Question 8 results by vote calculating the sum and percentage (this can all be done in the Socrata User Interface). I could not find the integer vote choice definitions, therefore I had to infer the values from the summary information in the PDF report. Excluding rounding errors, the results are the same, as can be seen in the Social Data Player embedded below:

For visualization choices, I plotted Coons vs Castle on total voter favorability:

Chis Coons charted against favorability by party:

Disclaimer: I have no affiliation or preference for any candidate in any State of Delaware political party. My goal for this write-up was to demonstrate how to visualize and share political data. If you have any questions on how to do any of these visualizations, please submit via @toddstavish.

Python Arduino Serial Port Text Communication (Send from PDE; Receive via PySerial)

Test code to make sure Python and Arduino are working properly. The PDE sends a string out of the serial port. The Python code uses pyserial to receive it and then prints the message to standard out. Pick it up at GitHub via “git clone git://github.com/toddstavish/Python-Arduino-Serial-Text-Send-Receive.git” Any questions or suggestions, you can find me here. 

Country Spreadsheet List With Latitude, Longitude, ISO 3166 Codes, and Flag Images

Spreadsheet to help create heatmaps in Socrata or Google geomap visualizations.

Spreadsheet download:

Spreadsheet with flag images (Countries.xls)

Google Docs:

Country List Latitude Longitude ISO 3166 Codes

Socrata:

Country List ISO 3166 Codes Latitude Longitude

Scribd:

Country List Latitude Longitude Flags ISO 3166 Codes

Previews:

Country List Latitude Longitude Flags ISO 3166 Codes