AnyCloud Bluetooth Advertising Scanner (Part 5)

Summary

In this article I will add a new task to the AnyCloud BLE Advertising Scanning application which will save the advertising data into a database.

Story

There is still a boatload of mostly unintelligible advertising data coming ripping onto our screen.  It is FINALLY time to start fixing that.  In this article I will create a new task called the advertising database task which will hold the history of advertising packets that I have seen.  I will update the Bluetooth Manager task to submit the advertising packets to a queue running in the advertising database task.

For this series of articles I will:

  1. Discuss an overview of Bluetooth Application Architecture (Part 1)
  2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
    1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
    2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
    3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
    4. Build the Basic Project (Part 2)
    5. Explain BLE Advertising - Scanner/Observer (Part 3)
    6. Explain BLE Advertising - Advertiser (Part 3)
    7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
    8. Add Observing to our Project (Part 3)
    9. Program and Test the Project (Part 3)
    10. Add the IoT Expert Bluetooth Utility Library (Part 4)
    11. Mulithreading Dangers (Part 4)
    12. Add a Queue to the Bluetooth Manager Thread (Part 4)
    13. Add a Timer to Process the Queue (Part 4)
    14. A Potential Threading Bug (Part 4)
    15. Add a Scan Off & On Command (Part 4)
    16. Create an Advertising Data Database Task (Part 5)
    17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
    18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
    19. Create an Advertising Data Database (Part 5)
    20. Add a Command to Print the Database (Part 5)
    21. Refactor a silly duplication (Part 6)
    22. Update the print (Part 6)
    23. Add the "decode" command (Part 6)
    24. Add the decode packets functionality (Part 6)
    25. Program and test (Part 6)
    26. Design a new advertising database (Part 7)
    27. Update to the new database (Part 7)
    28. Update printing (Part 7)
    29. Update the advertising add (Part 7)
    30. Create a watch command (Part 7)
    31. Create a record command (Part 7)
    32. Create an erase command (Part 7)
    33. Update the filter data structure (Part 8)
    34. Fix the Add function (Part 8)
    35. Add the filter command (Part 8)
    36. Update the printing (Part 8)
    37. Fix a memory leak (Part 9)
    38. Add packet age (Part 9)
    39. Fix the printing, again (Part 9)
    40. Fix the add packet function (Part 9)
    41. Update the database structure for sort and purge (Part 10)
    42. Add the sort and purge commands (Part 10)
    43. Add the sort functionality (Part 10)
    44. Add the purge functionality (Part 10)
    45. The End & Commentary (Part 10)

    All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

    There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

    Create an Advertising Data Database Task

    We need to create the file advertisingDatabase.h which will hold the task prototype (so that main can get going).

    Then create the advertisingDatabase.c to hold the actual database code.  It will start with the definition of messages which can be sent to the task.  For now just “ADB_ADD”.  To make things a little bit simpler these command can have two data elements (which I call data0 and data1).  Then the main part of the task just

    1. Creates the queue to manage the messages
    2. Process the message until the end of time

    To start the task, you need to add it to main.c.

    When I build and program this, you can now see the new task.  Good that working.

    Update the Advertising Database to Accept Submitted ADV Packets

    If you recall our original setup was to take advertising packets in the Bluetooth Manager thread and print out the data.  The first thing that we want to fix up is the ability of the advertising database task to accept advertising packets which are pushed to its command queue.   To prepare for this I create two local variables to hold the data.

    Then I update the ADB_ADD command.  My first, and really simple fix, is to grab the printing code from the Bluetooth Manager task.  Obviously this won’t be an improvement from the original program as far as the users goes, but it will verify that the tasks are working properly together.

    Then I add a command to the advertisingDatabase.h which the Bluetooth Manager task can call to submit advertising packets

    The actual command in advertisingDatabase.c just takes the advertising information, puts it in a command message, then submits it to the command queue.

    Update the Bluetooth Manager to Submit Adv Packets

    Now I go and edit the bluetoothManager. c to submit packets rather than print them.  To do this I greatly simplify the callback.  There is one VERY important issue to deal with, which is one of those potential religious war issues.  Memory.

    When you get the callback from the stack, it gives you POINTERS to data for the advertising packet that reside inside of buffers inside of the stack.  As soon as this callback returns this memory is purged.  To prevent this data from getting cleaned up by the stack I

    1. Malloc some memory for the wiced_bt_ble_scan_results
    2. Malloc some memory for the advertising data
    3. Make a copy of the data
    4. Submit it to the Advertising Database

    I KNOW from the spec that the largest data packet is 31-bytes (actually it is 31-bytes + one more field with length 0).  So I know the maximum length is 32-bytes  This means that in many situations I will be copying GARBAGE into my buffer if the packet is less than 32 bytes long.  I think that this is simpler than calculating the length and then only copying that much data.

    When I run this updated program I should get the same stream of data coming out on the serial port.  Sure enough the new thread is working.

    Create an Advertising Data Database

    Now, lets create an actual database.  To simplify things my database is just an array of structures.  One structure per bluetooth device.  The structure will contain a pointer to the information about the device it just saw and the actual raw data.

    Then I will create several helper functions to work with the database

    1. Find devices in the database given a mac address
    2. Print an entry in the database
    3. Add entries to the database

    First, find an entry in the database.  This function will search through the database and compare the mac address against the mac address in the database.  When the memcmp ==0 meaning it found a match, it will return that entry.

    The print function will make sure that you asked for a legal entry (much must be greater than 0… and less than the max).  Then it will print out the mac address and the raw data.  In a future post I will add a smarter print out.

    To add an entry to the database, first make sure that it isn’t already in the database.  Then when you are sure that it isn’t the database, you just add the pointers to your table.  You need to make sure and not go beyond the end of the table, and if you did, you will have effectively blown away the last entry in the table.  Oh well.

    Add a Command to Print the Database

    Now we want to add the ability to print from the command line.  So add a new command message to the list of legal commands.

    Then create a new function to print.  If you send in a “-1” it will print the whole table.  Otherwise just print the individual entry.

    Now edit usercmd.c to have the new command line.  Notice that I use “sscanf” which obviously has some issues.  Too bad.

    When I program the project it immediately prints out a bunch of devices that are at my house.  Then you can see I run the “print” command which prints the table.  Finally P do a print 0 to just print the first entry.

    In the next article I will add smarter diagnostics to the advertising packets.

    AnyCloud Bluetooth Utilities Library

    Summary

    This article is a discussion of a library of utilities functions that support AnyCloud Bluetooth development.  It includes settings to configure the hardware, functions to decode stack events, functions to decode advertising packets etc.

    Story

    The Cypress, now Infineon, Modus Toolbox team has been working to bring the WICED Bluetooth devices to be closer and more compatible with the PSoC 6 tools.  One of the important enablement things we have done is turn on the WICED Bluetooth Host stack on the PSoC 6 so that it can effectively talk to the CYW43XXX Bluetooth / WiFi combo chips.

    As I have been using all of the new stuff I found myself adding my own custom functionality…. and after a while I realized (finally) that I should put all of those little helper things into a library, specifically an IoT Expert library.  For now this library contains:

    • The Bluetooth Platform Configuration Settings
    • Functions to decode stack events
    • Functions to decode advertising packets

    but imagine with time I will add more functions and templates.

    bt_platform_cfg_settings

    As I discussed in the article AnyCloud Bluetooth Advertising Scanner (Part 1), every single AnyCloud BLE Stack project will require a structure of type wiced_bt_cfg_settings_t.  This structure is used to setup the hardware before getting the stack going.  Remember you need to make a call like this to initialize the Bluetooth hardware.

    At some point this file will be included automatically for you by the Modus Toolbox team, but for now I have added this to my library.  This file look like this.  Basically a bunch of pin definitions which are set for you automatically by the BSP.  Obviously you can make your own, but this should work on all of the Cypress development kits (I think)

    btutil_stack

    When you startup the Bluetooth Host stack you are responsible for providing a “management callback” which has the function prototype

    The first parameter is an “event” which is  just an enumerated list of possible events by the Bluetooth Host Stack.  Here is the actual list.

    While you are trying to figure out what is going on during the development, it is very useful to be able to print out the name of the events (instead of the numbers).  In other words instead of doing

    it is way better to do

    While I was looking at an example project I found this function which I thought was awesome.

    And then I learned something new when I looked at the “function” CASE_RETURN_STR which used compiler trick to turn an enumerated value into a string.

    This allows you to do this in your management callback (notice line 16 where the string is printed out)

    It turns out that there are several other places where the stack gives you an event-ish thing.  So these functions were created as well

    btutil_adv_decode

    In AnyCloud Bluetooth Advertising Scanner (Part 3) I discussed the format of the advertising packet.  So I created functions which will decode the data in the advertising packets.  More on the future article AnyCloud Bluetooth Advertising Scanner (Part 4 or maybe 5 or maybe 6).  Here are the functions:

    btutil

    And because I like to have just one include to get access to all of the function I created “btutil.h” which just includes all of the headers in one place.

    Add to the IoT Expert Manifest

    In order to set it up for my library to live in the library manager I updated the IoT Expert Manifest file to have a link to the GitHub repository

    And now the library manager has the IoT Expert Bluetooth Utilities.

     

     

    AnyCloud Bluetooth Advertising Scanner (Part 1)

    Summary

    The first of several articles discussing the use of the AnyCloud BLE Stack to build advertising scanner/observers.

    Story

    A few summers ago while I was writing the WICED Bluetooth Academy book, I created a WICED based BLE advertising scanner.  Actually, I created a framework for the scanner and the remarkable summer intern I had finished the work.  That project has been sitting in the source code repository for the Bluetooth class, mostly only shown to face-to-face students.  This scanner is built using some of the original code combined with the new AnyCloud Bluetooth SDK.  It will act sort-of-like LightBlue or one of the other Bluetooth advertising scanners you might run on your phone, but with a serial console.

    Sometime in the last few months we released the Bluetooth SDK for AnyCloud (things have been crazy and I have lost track of time)  This SDK has all of the stuff needed to add Bluetooth to your AnyCloud project using one of the Cypress Bluetooth/WiFi combo chips.  I had not had a chance to try it out, so I decided to build a Bluetooth project and then port the scanning code.

    For this series of articles I will:

    1. Discuss an overview of Bluetooth Application Architecture (Part 1)
    2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
      1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
      2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
      3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
      4. Build the Basic Project (Part 2)
      5. Explain BLE Advertising - Scanner/Observer (Part 3)
      6. Explain BLE Advertising - Advertiser (Part 3)
      7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
      8. Add Observing to our Project (Part 3)
      9. Program and Test the Project (Part 3)
      10. Add the IoT Expert Bluetooth Utility Library (Part 4)
      11. Mulithreading Dangers (Part 4)
      12. Add a Queue to the Bluetooth Manager Thread (Part 4)
      13. Add a Timer to Process the Queue (Part 4)
      14. A Potential Threading Bug (Part 4)
      15. Add a Scan Off & On Command (Part 4)
      16. Create an Advertising Data Database Task (Part 5)
      17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
      18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
      19. Create an Advertising Data Database (Part 5)
      20. Add a Command to Print the Database (Part 5)
      21. Refactor a silly duplication (Part 6)
      22. Update the print (Part 6)
      23. Add the "decode" command (Part 6)
      24. Add the decode packets functionality (Part 6)
      25. Program and test (Part 6)
      26. Design a new advertising database (Part 7)
      27. Update to the new database (Part 7)
      28. Update printing (Part 7)
      29. Update the advertising add (Part 7)
      30. Create a watch command (Part 7)
      31. Create a record command (Part 7)
      32. Create an erase command (Part 7)
      33. Update the filter data structure (Part 8)
      34. Fix the Add function (Part 8)
      35. Add the filter command (Part 8)
      36. Update the printing (Part 8)
      37. Fix a memory leak (Part 9)
      38. Add packet age (Part 9)
      39. Fix the printing, again (Part 9)
      40. Fix the add packet function (Part 9)
      41. Update the database structure for sort and purge (Part 10)
      42. Add the sort and purge commands (Part 10)
      43. Add the sort functionality (Part 10)
      44. Add the purge functionality (Part 10)
      45. The End & Commentary (Part 10)

      All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

      There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

      Bluetooth Application Architecture

      Bluetooth applications are divided into these four pieces

      1. You user application which responds to events and sends messages from/to the Bluetooth host stack
      2. A Bluetooth Host Stack
      3. A Bluetooth Controller Stack
      4. The Bluetooth Radio

      These four pieces can be divided into multiple chips, as few as one or as many as four.  However, for this article, the project will be built to run on a PSoC 6 + CYW43012 WiFi/Bluetooth Combo chip.  Specifically:

      1. My scanner application running on the PSoC 6
      2. The Bluetooth Host Stack running on the PSoC 6
      3. The BlueTooth Controller Firmware running on the CYW43012
      4. A Bluetooth Radio on the CYW43012

      But how do they talk?  Simple, there is:

      1. A UART Host Controller Interface (HCI) between the two chips
      2. A GPIO to serve as a deep sleep wakeup from the 43012 –> PSoC 6
      3. A GPIO to serve as the bluetooth controller wakeup from the PSoC 6 –> 43012
      4. A GPIO to turn on the Bluetooth regulator from the PSoC 6 –> 43012

      Here is the block diagram from the CY8CKIT-062S2-43012 Kit Guide.  Notice that signals labeled UART and Control going between the PSoC 6 and the CYW43012.

      And when you read more deeply into the schematic you can see the signals labeled

      • BT_UART_TXD/RXD/CTS/RTS
      • BT_HOST_WAKE
      • BT_DEV_WAKE
      • BT_REG_ON

      How to Start the AnyCloud Bluetooth Stack

      To actually start the AnyCloud Bluetooth stack you will call two functions

      1. cybt_platform_config_init – that will setup the hardware interface to the CYW43012
      2. wiced_bt_stack_init that will:
        1. Start a task to manage the Host Controller Interface
        2. Download the controller firmware to the CYW43012
        3. Start a task to manage the host stack
        4. Initialize both the host and the controller
        5. Call you back when that is all done

      Here is an example from main.

      When you look at these two function calls, you will find that you need to provide three things:

      1. A platform hardware configuration structure called bt_platform_cfg_settings
      2. The Bluetooth stack configuration settings structure called wiced_bt_cfg_settings
      3. A management callback called app_bt_management_callback

      bt_platform_cfg_settings

      The purpose of the hardware configuration structure is to define the UART + parameters and the wakeup GPIOs.  Specifically, the hardware configuration structure defines the configuration of the host controller interface (hci)

      1. The HCI transport scheme (in this case UART)
      2. The pins of the UART
      3. Baud Rate
      4. Data Bits
      5. Stop Bits
      6. Parity
      7. Flow Control

      And the controller low power behavior (in the .controller_config member)

      This is a fairly standard configuration and I think that we should help you by providing this structure somewhere in the BSP.  But for now, you need to provide it (in an upcoming article I’ll update the IoT Expert Bluetooth Library to provide it).  Here is the specific structure that I will be using.

      wiced_bt_cfg_settings

      The Cypress WICED Bluetooth Stack has a boatload of configuration settings.  When you call the stack start function you need to provide all of those settings in a structure of type “wiced_bt_cfg_settings_t” which is actually a structure of structures.  There are several basic ways to set these settings

      • Start from scratch and build you own settings
      • Copy from an example project
      • Use the Bluetooth Configurator to generate the structure

      For the purposes of THIS project I started by copying the structure from on of the example projects and then modifying the three numbers that were relevant to me.  Specifically

      • max_simultanous_link – which I changed to 0 because this is simply a Bluetooth Observer
      • low_duty_scan_interval – how long in the window to listen for advertising packets
      • low_duty_scan_window – how wide the window of listening should be

      app_bt_management_callback

      The last thing that you need to provide is a management callback.  This function is called by the Bluetooth Stack when a “management event” occurs.  There is a big-long-list of enumerated events of type wiced_bt_management_event_t.  The events include things like the the stack started “BTM_ENABLED_EVENT”.  Each event may have data associated with the event which is passed to you in a pointer to wiced_bt_management_event_data_t.

      You typically deal with these events with a giant switch statement like this:

      Tasks

      The Bluetooth stack on the PSoC6 is operated with two tasks.  Specifically, when you call the wiced_bt_stack_init it will startup:

      1. CYBT_HCI_Task – a task that sends and receives HCI packets going to the Radio chip
      2. CY_BT_Task – a task that manages the Bluetooth Host Stack

      Here is print of the task list from my project:

       

      Now with the background in place, in the next article I will discuss Bluetooth advertising and how to build the observer project.

      ModusToolbox 2.2 Template Project – FreeRTOS + NTShell

      Summary

      This article discusses the new library structure that was released with ModusToolbox 2.2.  I explain it by showing the creation of a template project that use FreeRTOS and NT Shell.

      Story

      I have often started projects from the IoT Expert FreeRTOS template project.   I realized the other day that almost always the first thing I do after creating the project is add the NT Shell library.  My friend Hassane has a personal mantra that if he is going to do the same job more than once he will always automate it.  I should have listened to him on this one because I have done it a bunch of times.

      In Modus Toolbox 2.2 we have created a new library scheme which allows sharing of libraries between projects.  So this will also be a good example of how that works.

      This will also give you another example of adding template projects to your own manifest.

      Here is what I am going to do:

      1. Create a project from the IoT Expert FreeRTOS Template
      2. Add the NTShell Library & Examine New Library Structure
      3. Update the Project and Program
      4. Add the Task List functionality (a nice feature of FreeRTOS)
      5. Put the new template on GitHub
      6. Update the IoT Expert App Manifest
      7. Test the new Template

      Create & Test a project from the IoT Expert FreeRTOS Template

      I will start the whole process by creating  new project using my existing base template.  The kit that I happen to have on my desk right now is the CY8CKIT-062S2-43012.

      Select the IoT Expert FreeRTOS Template and give it a name.  Notice that I add “NTShell” to the name (because that is what Im gonna add)

      When you click create, Modus will do its magic and build you a complete project.

      Today Im going to edit using Visual Studio Code.  Actually almost always I edit using Visual Studio Code.  You can do all of these tasks using Eclipse as well.  To turn my created project into a VSCODE project run “make vscode”

      Before getting to far down the road I like to run a build to make sure everything is OK.  So “make -j build”

      Then program it, just to make sure.  “make program”

      Add the NTShell Library & Examine New Library Structure

      Everything is working and my basic project has FreeRTOS and a blinking LED.  Now let’s add the NT Shell Library.  To do this run the library manager by running “make modlibs” (or click on the button in Eclipse).  Select Libraries –> IoT Expert –> ntshell

      When you press update, the library manager will do its thing again.

      When I look in the “deps” directory, I see some new file types called “.mtb”.  These files tell your project where to find each of the libraries.  Notice that the middleware-ntshell.mtb points to “$$ASSET_REPO$$”.  Where is that?

      If you have a aook at the Makefile it tell you that it is “../” and “mtb_shared”.

      To start editing I will run Visual Studio Code by typing “code .”.  The first thing that happens is that it notices that this is a workspace instead of just a directory.

      And when you look at the workspace you can see that it knows about both the example project as well as “mtb_shared”.  That is sweet.

      Update the Project and Program

      Now follow the instructions from the middlware-ntshell readme by copying “usrcmd.*” into my project.

      Then I edit main.c to

      1. #include “usrcmd.h” on line 8
      2. Start the shell task which is called “usrcmd_task” on line 39

      Now buid/project by running “make -j program”

      Now I have a functional shell.  Here is the serial console.

      Add the Task List functionality

      In the main.c above I started up the usrcmd_task with a stack size of config_MINIMAL_STACK_SIZE * 4.  Where did I get that?  Well the first couple of times I did this it crashed by running out of stack so I tried bigger and bigger numbers until it stopped crashing.  This is a kind of a pain in the ass.   If you know FreeRTOS there is a function called “vTaskList” which will give you stats about the tasks.

      In order to use that function you need to turn it on.  Notice that I #ifdef and #if to see if it is turned on inside of usrcmd.c

      So let’s turn it on by editing “FreeRTOSConfig.h” and enabling the two required defines.

      Now build/program.

      When I run help you can see I have a new command called “tasks” which lists all of the tasks and their stack high water marks.

      Put the Template on GitHub

      I am happy with my new template.  So, I go to GitHub and create a new repository.

      Then on my current project:

      1. I blow away the git history (didnt really have to do that).
      2. Create a new git repo “git init .”
      3. Add a pointer to GitHub “git remote add….”
      4. Add all of the files “git add *”
      5. Add the .gitignore “git add .gitignore”
      6. Commit the changes “git commit…”
      7. Push it to GitHub “git push …”

      Update the Manifests

      To get access to the new template I need to add it to the IoT Expert App Manifest.  I edit the xml file to have the new app

      Now I need to git add, git commit and git push it to GitHub.

      Test the new Template

      Everything should be working so make a new project.

      Cool.  There is the new template.

      When I program it… everything is cool.

      And the project seems to be doing the needful.

      PSoC 6 SDK OneWire Bus (Part 5): Round out the OWB driver

      Summary

      This article shows the completion of the PSoC 6 SDK One Wire Bus library.  It shows the test apparatus for evaluating DS18B20 sensors.

      Story

      If you remember from the previous articles, I created the one wire bus library by looking at the function calls in the DS18B20 library and reverse engineering the function prototypes.  And, as I said in the earlier article, it would have been way better to just look at the David Antliff “OWB” library.  But that is not how it went down.  I wonder how much time in the world is wasted by programmer re-implementing code that already exists?

      After the first four articles, I had three functions which the DS18B20 library defined, but I had not yet implemented.

      I had not implemented them because I was not totally sure how they worked.  So, I decided to go back to GitHub and see what David had done originally.  When I got to the GitHub site, https://github.com/DavidAntliff/esp32-owb/blob/master/include/owb.h, there were actually quite a few functions in his owb library that I had not implemented.

      Here is the list:

      In this article I will create and/or copy the missing functions.  As I look through his implementation I also notice that we have some style differences that I will discuss.  In general I will say that his implementation is very good and any place that I did something different was just a matter of programming taste.

      I was origi