The Creek: Creek Server 1.2

In the previous post, I talked about two problems that I had discovered in my system.  I addressed first one,  a serious bug in the data collection, last time.  In this post Ill address the slow performance of the website.  A common way to solve a performance problem is with a cache.  Instead of running the web page charts in real time, I will create them automatically every couple of minutes, then serve up a cached version.   In some ways this kind of sucks.  If I had known that I was going to go this way then there would have been 0 need of Tomcat and the Java Server Pages.  Oh well.

I start this process by copying the CreekHistory java object into the “getCreek” part of my project.  I then modify the main batch Java program, called “CreekServer” to understand the CreekHistory object.

Screen Shot 2016-05-08 at 11.19.48 AM

Then I add a new function “createHtml” to the CreekHistory object.  This function creates the webpage with the current information on it.  Basically

  1. A table of the last three hours [line 221-235]
  2. Link to the current chart [line 237]
  3. Link to the floods charts [line 237/238]

Screen Shot 2016-05-08 at 11.26.31 AM

I then add the “Current” command to the “runi2c” batch program.

Screen Shot 2016-05-08 at 11.39.43 AM

This program is run every 2 minutes by the crontab on the Raspberry Pi.  I picked 2 minutes because that is a good bit longer than each job takes to run in total.   Notice that I check to see if the jobs are already running and quit the “runi2c” job if the chart creation process is already going on.

At this point I have (at least) several things which are ugly

  1. I don’t have a good back up system
  2. I duplicate the “CreekHistory” object in the JSP as well as the “getCreek”
  3. I duplicate the “readProperties” method into a bunch of different objects
  4. I have 5Vs (I think) on the I2C pins of the Raspberry Pi, which I don’t understand why this isn’t a problem
  5. I need to get the PSoC4 Bootloader Host going on the Raspberry Pi

Index Description
The Creek: IOT for the Elkhorn Creek Introduction
The Creek: Solution Architecture 1.0 Overall architecture
The Creek: Creek Board 1.1 Eagle layout of the board
The Creek: Creek Board 1.0 – RCCA A discussion of the errors in the 1.0 board
The Creek: CYPI, a Raspberry Pi to Arduino Bridge PSoC4 <--> Raspberry Pi Bridge Board
The Creek: PSoC4 Creator Schematic and Firmware Firmware to interface with the temperature and pressure sensors
The Creek: Testing the Firmware Using tools to verify that the PSoC 4 Firmware is working correctly
The Creek: Testing the Bootloader Make sure that you can load new firmware into the PSoC
The Creek: Software Architecture All of the Raspberry Pi software connections
The Creek: Install MySql Instruction to configure MySql
The Creek: Install Tomcat Instruction to configure Tomcat JSP Server
The Creek: Data Collection Java (Part 1) The Java program that reads the I2C and saves it in the database
The Creek: Data Collection Java (Part 2) The Java program that reads the I2C and saves it in the database
The Creek: Create the Chart with JFreeChart Using open source Java charting software to create plots of the Creek Depth
The Creek: Flood Event Data Processor A batch program to create analyze the database and create a table of flood events
The Creek: Flood Event Web Page A batch program to create the flood event web page
The Creek: Creek Server 1.1 Updates to all of the back off server programs to integrate charts
The Creek: JSP Web Page for www.elkhorn-creek.org The JSP program to make the table and display the website
The Creek: Raspberry Pi Clock Stretching Sorting out a bug in the system having to do with the Broadcomm Raspberry Pi Master not functioning well with clock stretching
The Creek: Creek Server 1.2 Caching the web pages to make them faster

 

WICED WiFI

Last week I had the supreme privilege of hosting the WICED WiFI + Bluetooth + Zigbee software team at my office in Kentucky. This included the overall manager for WICED software (a truly remarkable guy), the engineering managers for WiFI and Bluetooth, the head of applications for WICED as well as a bunch of the firmware guys.  It occurred to me during the week that the people joining Cypress was the best part of the Broadcomm IOT acquisition.  And that is saying something as I really like the products.  Also at the summit were all of the software engineering leaders for PSoC (who I have worked with closely all of my career).  Needless to say, it was a bunch of badass developers.

The purpose of the meeting was to introduce the PSoC team to WICED and then talk about the future roadmap for those products.  Obviously I can’t talk to much about the 2nd part… well actually the only thing I can say is that it will be amazing as we will be able to offer PSoC with the power of WICED.

What I can talk about is the first part.  So, I thought that I would show you one of the things that we did with WICED.  First of all, WICED (Wireless Internet Connectivity for Embedded Devices) is the brand name that Cypress uses to describe all of the WiFi, Zigbee and Bluetooth chips and modules that were acquired from Broadcom/Avago.  The other thing we call WICED is the WICED SDK which is used to mean Eclipse plus all of the tools (programer, plugins etc) plus the software library that is used to build products using the WICED chips and modules.

In the world of programming the first example is always “hello world”.  In the world of MCUs the first example is always “blinking led”.  It turns out that the first example in WiFi is “scan” to show that you can see all of the WiFi networks around you.   The purpose of all of these examples is to prove that all of the tools can do their thing.

To start with they gave me this devkit, the BCM94343WWCD1_1  IMG_3049

The first thing to do is install WICED 3.7.  When you start WICED you will see a screen like this:

Screen Shot 2016-08-22 at 8.22.06 AM

For the purposes of the first design there are two important things to see on this screen.  First on the left side is the project explorer.  It has all of the guts of WICED.  As part of the installation we provide you a bunch of “apps”.  These apps range from simple examples (in the snip folder) to full fledged production quality applications (in the demo folder)

  • demo – full fledged applications
  • snip – short example projects
  • test – tools for debugging and testing wifi
  • waf – WICED Application Framework support (like an OTA Bootloader)
  • wwd – low level driver examples

Screen Shot 2016-08-22 at 8.27.04 AM

The example that I want to build is “scan”  specifically “scan.c”.  That can be found in the apps/snip/scan folder.  In this screenshot you can see that I opened “scan.c”

Screen Shot 2016-08-22 at 8.43.49 AM

The next thing that you need to do is build a “make target”.  The WICED team built a makefile that can seemingly do everything.  The makefile uses the name of the make target to setup all of the options required to do the make.  If you look on the right side of the screen you can see the currently existing targets:

Screen Shot 2016-08-22 at 8.24.40 AM

The easiest way to make a new target is to copy/paste a currently existing target.  Then you can right click on the new target and edit it to get things setup correctly.  The target name defines:

  • snip – the directory
  • scan – the subdirectory will the files (scan.c and scan.mk)
  • BCM94343WWCD1 – the name of the devkit (you can see it on the picture of the devkit)
  • download run – instructions to go ahead a boatload and start the app running

Screen Shot 2016-08-22 at 11.32.06 AM

Next, I plug in the devkit.  When it attachs, the devkit will enumerate as two USB devices

  • WICED USB JTAG Port
  • A serial port (in this case on COM12)

Screen Shot 2016-08-22 at 11.29.45 AM

After I plug in the kit I run Putty and attach to COM12 at 115200 baud

Screen Shot 2016-08-22 at 11.36.32 AM

And finally, double click the make target to build, download and run.  After it starts, the Putty screen fills up with all of the WiFI networks that are around me.

Screen Shot 2016-08-22 at 11.30.39 AM

All of that was pretty easy to get going.  Next lets see if I can actually do something.  Last week I showed the guys the Elkhorn Creek Water Level monitoring project and I told them that by the end of this week I would put one of their devkits into that system, so the next several posts will be about that process. (I hope)

The Creek: Web Pages with Java Server Pages (JSP)

The last piece in the Elkhorn Creek IOT puzzle are the webpages that let you view the creek data on the internet.  There are two pages, one with a table of the data and a chart and the other a dump CSV file of the database.  When it came time to work on this post, I realized that I had little memory of the evils deeds I had originally done to make the JSP pages work.  As I sorted through the code that was existing on the current production Raspberry Pi environment, all I could see was an undocumented mess.

Making dynamic web pages is ugly business.  As best I can tell, there is no clean way do this job (though I am planning on showing some ways to make it better in future posts).  A JSP is an interleaved mashup of HTML and Java.  To make a JSP, you create a file with sections of both languages separated by markers.   In the following example you can see an example that rolls two dice (generates a random number between 1-6).  The sections of code that are Java are marked with a “<%” and a “%>”.

<html>
<head><title>Roll the dice</title></head>
<body>
<%
int die1 = (int)(Math.random() * 6 + 0.5);
int die2 = (int)(Math.random() * 6 + 0.5);
%>
<h2> Die1 = <%= die1 %></h2>
<h2> Die2 = <%= die2 %></h2>

<a href=”<%= request.getRequestURI() %>”><h3>Try Again</h3></a>
</body>
</html>

When a web request is made, the Tomcat JSP Webserver takes the page (in JSP format)

  1. Converts it to Java (turns raw HTMLs into java commands e.g. “<html>” turns into “out.println(“<html>”);”
  2. Runs the compiler
  3. Executes the java program and pipes the output back to your web browser

To makes things go faster Tomcat will cache the output of 1+2 and only rerun it if something has changed.

Since I last worked on this system there is one thing that had greatly improved.  Specifically, this time I knew about Netbeans which is the Sun Microsystems IDE for editing Java.  Netbeans is cool for this project because:

  • It knows about Java (so it knows how to edit, highlight, search, refactor, syntax check etc)
  • It knows about JSP (so it knows how to edit…)
  • It knows how to edit Java Properties files
  • It has a JSP web server (GlassFish) integrated into the tool.  This allowed me to debug the web pages on my Mac (instead of on the RPi)
  • It knows how to make “war” files (the file format that you use to deploy a JSP web application)

The Netbeans project navigator lets you look at (and into) all of the files that are part of the JSP Web Application.  In the screenshot below you can see the organization of my project.  It has

  • A classes folder which will contain the compiled java classes.  It can also hold a “properties” file.  In this case I have a properties file called “config.properties” which contains the URL, User and Password for the mysql database
  • basic.jsp: The JSP file which displays the table and graph
  • index.html: a file which just loads the basic.jsp
  • CreekHistory.java: A java class to interact with the MySql Database.  This is essentially the “Model” of the MVC
  • A libraries folder with mysql-connector-java-5.1.6-bin.jar: the mysql library to connect to the database

Screen Shot 2016-05-03 at 7.13.26 AM

I like to use the Model View Controller (MVC) design paradigm of user interface programming.  In this methodology the model represents the data.  It knows nothing about user interfaces, how to create them, how to display them.  It is completely agnostic about GUI.  The view is a mechanism to display data. In this JSP web application, the view is just the raw html that is displayed on the web.  The last piece is the controller (aka the view controller).  It takes data out of the model and puts it into the view.  Here is a nice picture from Apples website (but if you google you will find lots of discussion about MVC programming)

model_view_controller_2x

The first thing that I created was Java class called CreekHistory.java.  This class has the responsibility to read the last 4 hours of depth and temperature data from the MySql database and put it into 15 minute buckets (using linear interpolation), and store the results into an array for easy retrieval by the controller.

In the first block of code (lines 29-31) I declare a few private variable that hold information out of the properties file about the database connection. The I declare a bunch of public variables which are the data interface to the model.

  • depth[] + temperature[] arrays of doubles that contain the depth and temperature in 15 minute buckets
  • depthDelta[] and temperatureDelta[] doubles that contain the change in depth and temperature since the start aka [0]
  • startTime a timestamp of the first datapoint (from the MySql database)

In the next clock of code (lines 43-46) I declare some constants (so that I don’t have magic numbers in my code).

On line 48 I declare a reference to a JspWriter.  This reference will allow me to print debugging information to the HTML output.  This is a clear violation of the MVC so I only use it for printing debugging information as it is a serious PITA to debug JSP code.

And finally the constructor which calls the function to read the properties file and then initialize the arrays.

Screen Shot 2016-05-04 at 6.25.48 AM

In the next section of code I have three helper functions.

The first function “interpolate” just performs a linear interpolation between two datapoints given known timestamps.

The “cToF” function just converts the a temperature from the bogus Centigrade units to useable Fahrenheit.

The last function, “readProperties” finds the config.properties file and reads the values into the global Properties class.  The only magic in this function occurs on lines 194-195 where it uses the ClassLoader object to search through the “classes” directory to find the file with the properties.

 

Screen Shot 2016-05-04 at 6.32.46 AM

The last function is “loadData” which has two sections.  The first section sets up and executes the MySql query to get the current Elkhorn Creek data.

Screen Shot 2016-05-04 at 6.40.44 AM

Once it has the data, it iterates through the data from the MySql database to find the datapoints that bracket the 15 minute buckets, then interpolates the results, then store it in the array.

Screen Shot 2016-05-04 at 6.51.08 AM

Finally the last part of the web server is the JSP page which outputs the actual HTML.

Lines 1-2 load two Java classes which I use on other sections of the page.  Specifically the Timestamp class which I use to do Time math, and the CreekHistory class which is the “model”

Lines 15-18 initialize the CreekHistory model object and tell it to load its data.  Notice on line 16 if I set the CreekHistory “out” public variable then it will put out debugging information.

Lines 24-37 just spit out an HTML table of the data from the CreekHistory model.

Screen Shot 2016-05-04 at 7.04.07 AM

That is all there is to it.  As with all of the code on this website you can get it from github.

Index Description
The Creek: IOT for the Elkhorn Creek Introduction
The Creek: Solution Architecture 1.0 Overall architecture
The Creek: Creek Board 1.1 Eagle layout of the board
The Creek: Creek Board 1.0 – RCCA A discussion of the errors in the 1.0 board
The Creek: CYPI, a Raspberry Pi to Arduino Bridge PSoC4 <--> Raspberry Pi Bridge Board
The Creek: PSoC4 Creator Schematic and Firmware Firmware to interface with the temperature and pressure sensors
The Creek: Testing the Firmware Using tools to verify that the PSoC 4 Firmware is working correctly
The Creek: Testing the Bootloader Make sure that you can load new firmware into the PSoC
The Creek: Software Architecture All of the Raspberry Pi software connections
The Creek: Install MySql Instruction to configure MySql
The Creek: Install Tomcat Instruction to configure Tomcat JSP Server
The Creek: Data Collection Java (Part 1) The Java program that reads the I2C and saves it in the database
The Creek: Data Collection Java (Part 2) The Java program that reads the I2C and saves it in the database
The Creek: Create the Chart with JFreeChart Using open source Java charting software to create plots of the Creek Depth
The Creek: Flood Event Data Processor A batch program to create analyze the database and create a table of flood events
The Creek: Flood Event Web Page A batch program to create the flood event web page
The Creek: Creek Server 1.1 Updates to all of the back off server programs to integrate charts
The Creek: JSP Web Page for www.elkhorn-creek.org The JSP program to make the table and display the website
The Creek: Raspberry Pi Clock Stretching Sorting out a bug in the system having to do with the Broadcomm Raspberry Pi Master not functioning well with clock stretching
The Creek: Creek Server 1.2 Caching the web pages to make them faster

 

The Creek: Creek Server 1.1

The last few weeks I have been building up the java programs that form the “back office” part of my system.  This is composed of

  • i2cb.java: A program that reads I2C registers of the Creek Depth and Temperature and inserts them into the creekdata database.
  • MakeChart.java: A program to create a PNG graph of Elkhorn Creek depth using JFreeChart
  • ProcessEvents.java: A program that searches the creekdata for “flooding events” and then inserts them into the flood event database.

Things worked OK, but I had several problems that I wanted to fix

  1. The MakeChart program was not using global properties file so I had duplicated things like the database password
  2. I had implemented myself into a design flaw (I wanted to chain information from i2cdb –> MakeChart –> ProcessEvents –> MakeChart
  3. I really wanted a single Java program with options to run all of the back office

The bottom line is this week will be spent fixing the design flaws and cleaning up the code.

The first thing that I did was create a new Java program called “CreekServer” which hooks all of the programs together and provides a unified command line interface.  This program looks at the first command line argument and decides to either:

  • Lines 12-15: print the help text
  • Lines 17-23: run the “MakeChart” function
  • Lines 24-29: run the ProcessEvents function, then update/create all of the historical flood charts [line 27]
  • Lines 30-34: run the GetData function

Screen Shot 2016-04-17 at 11.09.49 AM

The next block of code is a kludgey work around for my design flaw.  Specifically, after you have create a table full of the flood events you would like to create charts for each event.  However, I didn’t want to duplicate the chart object into the ProcessEvents class, so I decided to hack it into the system by adding a public interface to the ProcessEvents object.

  • [line 52] I tell the ProcessEvents object to read the floodevents table
  • [line 57-75] I iterate through all of the flood events that are in the public array of events (bad design)
  • [line 58-60] create the appropriate filename for the plot
  • [line 63] if the file doesnt exist (it was previously created) or the flood is ongoing (it has no end date) then create a new chart by calling the MakeChart object

Screen Shot 2016-04-17 at 11.32.27 AM

After I got done with the main program, I went back and fixed the ProcessEvents program to read its properties from the “config.properties” file.

Screen Shot 2016-04-17 at 11.49.54 AM

Finally, I fixed the properties problem in the MakeChart program:

Screen Shot 2016-04-17 at 12.02.08 PM

All of this code is available on the iotexpert github.

In the next post I am finally going to move into the JSP and Servlet code that builds the main site.

Index Description
The Creek: IOT for the Elkhorn Creek Introduction
The Creek: Solution Architecture 1.0 Overall architecture
The Creek: Creek Board 1.1 Eagle layout of the board
The Creek: Creek Board 1.0 – RCCA A discussion of the errors in the 1.0 board
The Creek: CYPI, a Raspberry Pi to Arduino Bridge PSoC4 <--> Raspberry Pi Bridge Board
The Creek: PSoC4 Creator Schematic and Firmware Firmware to interface with the temperature and pressure sensors
The Creek: Testing the Firmware Using tools to verify that the PSoC 4 Firmware is working correctly
The Creek: Testing the Bootloader Make sure that you can load new firmware into the PSoC
The Creek: Software Architecture All of the Raspberry Pi software connections
The Creek: Install MySql Instruction to configure MySql
The Creek: Install Tomcat Instruction to configure Tomcat JSP Server
The Creek: Data Collection Java (Part 1) The Java program that reads the I2C and saves it in the database
The Creek: Data Collection Java (Part 2) The Java program that reads the I2C and saves it in the database
The Creek: Create the Chart with JFreeChart Using open source Java charting software to create plots of the Creek Depth
The Creek: Flood Event Data Processor A batch program to create analyze the database and create a table of flood events
The Creek: Flood Event Web Page A batch program to create the flood event web page
The Creek: Creek Server 1.1 Updates to all of the back off server programs to integrate charts
The Creek: JSP Web Page for www.elkhorn-creek.org The JSP program to make the table and display the website
The Creek: Raspberry Pi Clock Stretching Sorting out a bug in the system having to do with the Broadcomm Raspberry Pi Master not functioning well with clock stretching
The Creek: Creek Server 1.2 Caching the web pages to make them faster