lithium flowers bloom » sensor network http://leejayxia.com/blog the weblog Sat, 15 May 2010 20:52:40 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 Sensor Network: Data Logging and Visualization http://leejayxia.com/blog/sensor-network-data-logging-and-visualization/ http://leejayxia.com/blog/sensor-network-data-logging-and-visualization/#comments Mon, 01 Mar 2010 07:50:18 +0000 L.J http://leejayxia.com/blog/?p=823 I’m working with Sebastian and Michael as “visualization” team for the sensor network assignment for this week’s Sociable Objects Workshop. It’s very challenging for the whole class to work on one single project, but so far it went on pretty well. Tasks are split and assigned and we also find ways to work with dependencies and such constraints.

Since we are just focusing on the visualization part, meaning that we take on only after all the data are aggregated at the coordinator xbee, our main effort will be data logging, mining and visualization. So the first thing we built is a pseudo data generator that feeds data in the same format as the data will be from the coordinator. We talked to the base station team and agreed that we will be getting raw API I/O RX packet and we will take care of the parsing and data storage.

I put together some simple arduino code to feed pseudo data, they could be easily replaced by a real coordinator and ideally we will not have to change the processing code on the other side which is listening to the serial channel.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
#define ADDRESS_COUNT 9
int addresses[9] = {
  0x0001,
  0x0002,
  0x0003,
  0x0004,
  0x0005,
  0x0006,
  0x0007,
  0x0008,
  0x0009,
};

void setup()
{
  Serial.begin(9600);
  randomSeed(analogRead(0));
}

void loop()
{
  for (int i = 0;i < ADDRESS_COUNT;i++)
  {
    if (random(1000) > 990)
    {
      if (random(1000) > 500)
      {
        sendPseudoPackage(addresses[i], 1);
      }
      else
      {
        sendPseudoPackage(addresses[i], 0);
      }
    }

  }
  delay(50);
}


void sendPseudoPackage(int address, int value) {  // pass either a 0x4 or and 0x5 to turn the pin on or off
  Serial.print(0x7E, BYTE); // start byte
  Serial.print(0x0, BYTE); // high part of length (always zero)
  Serial.print(0x17, BYTE); // low part of length (the number of bytes that follow, not including checksum)
  Serial.print(0x92, BYTE); // 0x92 is I/O RX Packet

  //05-12:  64-bit address
  Serial.print(0x00,BYTE);
  Serial.print(0x13,BYTE);

  Serial.print(0xA2,BYTE);
  Serial.print(0x00,BYTE);

  Serial.print(0x40,BYTE);
  Serial.print(0x33,BYTE);

  int low = address & 0xFF;
  int high = (address - low) >> 2;

  Serial.print(high, BYTE);
  Serial.print(low, BYTE);


  //13-14:  16 bit of recipient or 0xFFFE if unknown
  Serial.print(0xFF, BYTE);
  Serial.print(0xFE, BYTE);

  //15:  Receive Option
  Serial.print(0x01, BYTE);

  //16:  Num Samples
  Serial.print(0x01, BYTE);

  //17-18 Digital Channel Mask
  Serial.print(0x00, BYTE);
  Serial.print(0x01, BYTE); //only Digital I/O 0

  //19:  Analog Channel Mask
  Serial.print(0x00, BYTE);

  //20-21:  Digital Samples
  Serial.print(0x00, BYTE);
  Serial.print(value, BYTE);

  //no analog samples

  // checksum is all bytes after length bytes
  long sum = 0xFF; //fake
  Serial.print( 0xFF - ( sum & 0xFF) , BYTE ); // calculate the proper checksum
  delay(10); // safety pause to avoid overwhelming the serial port (if this function is not implemented properly)
}

Then I wrote a basic processing sketch which filters the data and stores only changes to the databases. The sketch itself servers as some sort of realtime visualization on the input, and we get a feeling of how often data are changing and potentially detects sensor failures if one sensor is not active for a very long time.

The processing sketch is sending data to a website that handles data logging. The website also provides query api that returns sensor history in json format, and functions like filtering by certain address. The site itself is lightweight and fairly easy to add new functionality.

The plan is to meet tomorrow and start building a web interface upon the api for the final visualization. Hopefully we could also get to verify if the pseudo data is identical to the real format, and maybe we can get a test run with the real setup.

Stay tuned!

UPDATE: A list of available API functions:

url: http://itp.nyu.edu/~lx243/sensornet/index.php/api/

all (meaning that the url will look like http://itp.nyu.edu/~lx243/sensornet/index.php/api/all)

returning all records in json format

find/:address

returning all records from address specified
address should be a string of 64bit address, e.g. 0013A20040330001

find_today/:address

find_by_day/:address/:day

day should be in the format of yyyymmdd, e.g. 20100301

status/:address

returning latest records from specified address

status_at/:address/:time

returning latest records by the time specified
time should be in the format of yyyymmddHHMMSS, e.g. 20100301140000

changes/:address

returning all changes at specified address

changes_today/:address

returning changes happened today at specified address

changes_by_day/:address/:day

returning changes happened today on specified date at specified address
day should be in the format of yyyymmdd, e.g. 20100301

stats/:address

returning open time, close time, changes, and total active time of the sensor

stats_today/:address

returning above stats from specified address

stats_by_day/:address/:day

returning above stats from specified address on specified day

UPDATE 2: Visualization

It took a while for Sebastian and I to make processing.js to talk to the API and get real time update. We used jQuery as a bridge to retrieve the data and feed it on the page to processing.js. But eventually all is fine:

]]> http://leejayxia.com/blog/sensor-network-data-logging-and-visualization/feed/ 0