AnyCloud Bluetooth Advertising Scanner (Part 5)

Summary

In this article I will add a new task to the AnyCloud BLE Advertising Scanning application which will save the advertising data into a database.

Story

There is still a boatload of mostly unintelligible advertising data coming ripping onto our screen.  It is FINALLY time to start fixing that.  In this article I will create a new task called the advertising database task which will hold the history of advertising packets that I have seen.  I will update the Bluetooth Manager task to submit the advertising packets to a queue running in the advertising database task.

For this series of articles I will:

  1. Discuss an overview of Bluetooth Application Architecture (Part 1)
  2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
    1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
    2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
    3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
    4. Build the Basic Project (Part 2)
    5. Explain BLE Advertising - Scanner/Observer (Part 3)
    6. Explain BLE Advertising - Advertiser (Part 3)
    7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
    8. Add Observing to our Project (Part 3)
    9. Program and Test the Project (Part 3)
    10. Add the IoT Expert Bluetooth Utility Library (Part 4)
    11. Mulithreading Dangers (Part 4)
    12. Add a Queue to the Bluetooth Manager Thread (Part 4)
    13. Add a Timer to Process the Queue (Part 4)
    14. A Potential Threading Bug (Part 4)
    15. Add a Scan Off & On Command (Part 4)
    16. Create an Advertising Data Database Task (Part 5)
    17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
    18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
    19. Create an Advertising Data Database (Part 5)
    20. Add a Command to Print the Database (Part 5)
    21. Refactor a silly duplication (Part 6)
    22. Update the print (Part 6)
    23. Add the "decode" command (Part 6)
    24. Add the decode packets functionality (Part 6)
    25. Program and test (Part 6)
    26. Design a new advertising database (Part 7)
    27. Update to the new database (Part 7)
    28. Update printing (Part 7)
    29. Update the advertising add (Part 7)
    30. Create a watch command (Part 7)
    31. Create a record command (Part 7)
    32. Create an erase command (Part 7)
    33. Update the filter data structure (Part 8)
    34. Fix the Add function (Part 8)
    35. Add the filter command (Part 8)
    36. Update the printing (Part 8)
    37. Fix a memory leak (Part 9)
    38. Add packet age (Part 9)
    39. Fix the printing, again (Part 9)
    40. Fix the add packet function (Part 9)
    41. Update the database structure for sort and purge (Part 10)
    42. Add the sort and purge commands (Part 10)
    43. Add the sort functionality (Part 10)
    44. Add the purge functionality (Part 10)
    45. The End & Commentary (Part 10)

    All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

    There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

    Create an Advertising Data Database Task

    We need to create the file advertisingDatabase.h which will hold the task prototype (so that main can get going).

    Then create the advertisingDatabase.c to hold the actual database code.  It will start with the definition of messages which can be sent to the task.  For now just “ADB_ADD”.  To make things a little bit simpler these command can have two data elements (which I call data0 and data1).  Then the main part of the task just

    1. Creates the queue to manage the messages
    2. Process the message until the end of time

    To start the task, you need to add it to main.c.

    When I build and program this, you can now see the new task.  Good that working.

    Update the Advertising Database to Accept Submitted ADV Packets

    If you recall our original setup was to take advertising packets in the Bluetooth Manager thread and print out the data.  The first thing that we want to fix up is the ability of the advertising database task to accept advertising packets which are pushed to its command queue.   To prepare for this I create two local variables to hold the data.

    Then I update the ADB_ADD command.  My first, and really simple fix, is to grab the printing code from the Bluetooth Manager task.  Obviously this won’t be an improvement from the original program as far as the users goes, but it will verify that the tasks are working properly together.

    Then I add a command to the advertisingDatabase.h which the Bluetooth Manager task can call to submit advertising packets

    The actual command in advertisingDatabase.c just takes the advertising information, puts it in a command message, then submits it to the command queue.

    Update the Bluetooth Manager to Submit Adv Packets

    Now I go and edit the bluetoothManager. c to submit packets rather than print them.  To do this I greatly simplify the callback.  There is one VERY important issue to deal with, which is one of those potential religious war issues.  Memory.

    When you get the callback from the stack, it gives you POINTERS to data for the advertising packet that reside inside of buffers inside of the stack.  As soon as this callback returns this memory is purged.  To prevent this data from getting cleaned up by the stack I

    1. Malloc some memory for the wiced_bt_ble_scan_results
    2. Malloc some memory for the advertising data
    3. Make a copy of the data
    4. Submit it to the Advertising Database

    I KNOW from the spec that the largest data packet is 31-bytes (actually it is 31-bytes + one more field with length 0).  So I know the maximum length is 32-bytes  This means that in many situations I will be copying GARBAGE into my buffer if the packet is less than 32 bytes long.  I think that this is simpler than calculating the length and then only copying that much data.

    When I run this updated program I should get the same stream of data coming out on the serial port.  Sure enough the new thread is working.

    Create an Advertising Data Database

    Now, lets create an actual database.  To simplify things my database is just an array of structures.  One structure per bluetooth device.  The structure will contain a pointer to the information about the device it just saw and the actual raw data.

    Then I will create several helper functions to work with the database

    1. Find devices in the database given a mac address
    2. Print an entry in the database
    3. Add entries to the database

    First, find an entry in the database.  This function will search through the database and compare the mac address against the mac address in the database.  When the memcmp ==0 meaning it found a match, it will return that entry.

    The print function will make sure that you asked for a legal entry (much must be greater than 0… and less than the max).  Then it will print out the mac address and the raw data.  In a future post I will add a smarter print out.

    To add an entry to the database, first make sure that it isn’t already in the database.  Then when you are sure that it isn’t the database, you just add the pointers to your table.  You need to make sure and not go beyond the end of the table, and if you did, you will have effectively blown away the last entry in the table.  Oh well.

    Add a Command to Print the Database

    Now we want to add the ability to print from the command line.  So add a new command message to the list of legal commands.

    Then create a new function to print.  If you send in a “-1” it will print the whole table.  Otherwise just print the individual entry.

    Now edit usercmd.c to have the new command line.  Notice that I use “sscanf” which obviously has some issues.  Too bad.

    When I program the project it immediately prints out a bunch of devices that are at my house.  Then you can see I run the “print” command which prints the table.  Finally P do a print 0 to just print the first entry.

    In the next article I will add smarter diagnostics to the advertising packets.

    AnyCloud Bluetooth Advertising Scanner (Part 4)

    Summary

    In this article I update the AnyCloud BLE advertising scanner to use the btutil library that was created in the previous post.  In addition, I add a command queue to the bluetoothManger and enable a new command to turn on and off scanning.

    Story

    If you have been following along until now, which I imagine that you have if you are reading this,  you will have gotten a vomit of device data blasting out onto your serial console.  This isn’t very helpful.  So now what?  I am going to divide this problem into two parts

    1. Creating a new user command to turn on and off scanning (this article)
    2. Creating a database to manage the data + a set of commands to dump it (next article)

    For this series of articles I will:

    1. Discuss an overview of Bluetooth Application Architecture (Part 1)
    2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
      1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
      2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
      3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
      4. Build the Basic Project (Part 2)
      5. Explain BLE Advertising - Scanner/Observer (Part 3)
      6. Explain BLE Advertising - Advertiser (Part 3)
      7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
      8. Add Observing to our Project (Part 3)
      9. Program and Test the Project (Part 3)
      10. Add the IoT Expert Bluetooth Utility Library (Part 4)
      11. Mulithreading Dangers (Part 4)
      12. Add a Queue to the Bluetooth Manager Thread (Part 4)
      13. Add a Timer to Process the Queue (Part 4)
      14. A Potential Threading Bug (Part 4)
      15. Add a Scan Off & On Command (Part 4)
      16. Create an Advertising Data Database Task (Part 5)
      17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
      18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
      19. Create an Advertising Data Database (Part 5)
      20. Add a Command to Print the Database (Part 5)
      21. Refactor a silly duplication (Part 6)
      22. Update the print (Part 6)
      23. Add the "decode" command (Part 6)
      24. Add the decode packets functionality (Part 6)
      25. Program and test (Part 6)
      26. Design a new advertising database (Part 7)
      27. Update to the new database (Part 7)
      28. Update printing (Part 7)
      29. Update the advertising add (Part 7)
      30. Create a watch command (Part 7)
      31. Create a record command (Part 7)
      32. Create an erase command (Part 7)
      33. Update the filter data structure (Part 8)
      34. Fix the Add function (Part 8)
      35. Add the filter command (Part 8)
      36. Update the printing (Part 8)
      37. Fix a memory leak (Part 9)
      38. Add packet age (Part 9)
      39. Fix the printing, again (Part 9)
      40. Fix the add packet function (Part 9)
      41. Update the database structure for sort and purge (Part 10)
      42. Add the sort and purge commands (Part 10)
      43. Add the sort functionality (Part 10)
      44. Add the purge functionality (Part 10)
      45. The End & Commentary (Part 10)

      All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

      There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

      Add the IoT Expert “btutil” Library

      Before we actually start all of the command queue stuff, lets move to the btutil library that I talked about in the previous post.  To do this, add the library using the library manager.

      Then delete bt_platform_cfg_settings.h and bt_platform_cfg_settings.c from your project.  Finally Rebuild and make sure that everything still works.  That is it.

      Multithreading

      Id like to explain that there is now some danger.  That danger comes from the fact that we have multiple tasks which are all accessing data plus functions that are talking to each other ASYNCHRONOUSLY.  Specifically we have:

      1. The Bluetooth Stack task – running the Bluetooth stack and management callback
      2. The Bluetooth Stack APIs – e.g. wiced_bt_ble_observe
      3. The usrcmd task – which is interacting with the user on the serial port and talking to the other tasks
      4. A timer_svc task – which runs software timers
      5. The advertising data (which I will start saving in the next article)

      When faced with this situation what I typically like to do is provide thread safe public functions for each of the tasks.  Then any other task can call these functions and know that things are not going to get corrupted by a race condition.

      To make the design thread safe, I typically like to put an RTOS Queue between the tasks.  These queues are a safe place to send and receive data in a “thread safe” way.  There are two basic design patterns that can be used

      1. Define a message structure (that gets pushed into the queue) and make it global (via a dot-h).  Define a queue handle and make it global (via a dot-h).  Then let any task build messages and push them into the queue to be received in the task that owns the queue.
      2. Define the message structure and queue.  Then define functions which are global (via a dot-h) which know how to interact with the queue.

      I typically think that the 2nd method is better, so that is what I am going to do here.

      1. In BluetoothManager.h I will provide a function called “btm_cmdScan”
      2. The usrcmd task will call the btm_cmdScan function which will
      3. Create a btm_cmdMsg_t with the “scan” command and data of true/false
      4. Then push it into the Bluetooth Manager Command Queue
      5. Where a timer callback in the Bluetooth Manager Task will take it out of the queue
      6. Figure out that it is a “scan” command
      7. Then will either turn on or off scanning

      Add a Queue to the Bluetooth Manager Thread

      So we need two things a message to push into a queue (just a structure) and we need a queue to push it into.  First the message which is just a structure with two elements.  The first element is a command and the second element is some data of type void.  The meaning of the void *data will be different based on the command.

      But how about the command?  The command is just an enumerate list of commands which will now start with just one command.

      And know we need to define the queue.

      Before you can use the queue you need to initialize it.  The best place to initialize this queue is in the management callback right after the stack gets going.  You can see that I tell FreeRTOS that there is a queue which can hold up to 10 commands.  I also tell it that each command is the sizeof the command message.

      Now we need to create a way for other tasks to create these command messages.  They will do this by calling a function which we will define in the bluetoothManager.h

      This function will live in bluetoothManager.c and it simply

      1. Creates a command
      2. Set the actual command to scan
      3. Sets the void* data to be enable … in other words start or stop scanning.  Remember that a void * can be anything.  See I cast a bool to a void *
      4. Finally push the data into the command queue

      Add a Timer to Process the Queue

      So now we have a method to push items into the queue.  How do we get them out of the queue?  To do that I will use a Bluetooth Stack timer that will run every 50ms.

      First, define the timer in bluetoothManager.c

      Then define a function which the timer will call.  This function will

      1. Try to get a message out of the queue
      2. IF there is a message it will use a big switch to look at the possible messages
      3. If the message is a scan
      4. Then call the wiced function to either start “observing” or stop “observing”

      The last thing you need to do is start the timer.  The best place to start the timer is in the management callback where you need to

      1. Create the timer
      2. Tell it to start and run every 50ms

      A Potential Threading Bug

      When I did the implementation originally I created what I thought was a threading bug.  Specifically I used the FreeRTOS timer to process the queue.  In other words instead of using a wiced_timer_ext_t I used a TimerHandle_t.  So what?

      The wiced_timer_ext_t is run INSIDE of the BluetoothStack task where the TimerHandle_t is run inside of the Timer_SVC task.

      So what?  I was afraid that the call to wiced_bt_ble_obsere was NOT thread safe and needed to be called inside of the same task as the stack.

      After some digging I found out that the Bluetooth Stack is threadsafe, so I worried for no reason.  Well, actually, you can never worry enough about making these kinds of threading bugs because they are viscously difficult to debug.

      Add a Scan Off & On Command

      The last thing that you need to do is add an actual command to the usercmd task to call the bluetooth manager function to turn on and off scanning.

      First, add a new prototype for your new command in usercmd.c.  Then add it to the list of legal commands.

      Then create the function to process the command line input and call the btm_scan function.

      Now build it and run it.  You should still get adv packets barfing all over your screen.  But now you can turn on and off the scanning with “scan on” and “scan off”.  In the next article we will create a database to hold the scan packets.

      AnyCloud Bluetooth Utilities Library

      Summary

      This article is a discussion of a library of utilities functions that support AnyCloud Bluetooth development.  It includes settings to configure the hardware, functions to decode stack events, functions to decode advertising packets etc.

      Story

      The Cypress, now Infineon, Modus Toolbox team has been working to bring the WICED Bluetooth devices to be closer and more compatible with the PSoC 6 tools.  One of the important enablement things we have done is turn on the WICED Bluetooth Host stack on the PSoC 6 so that it can effectively talk to the CYW43XXX Bluetooth / WiFi combo chips.

      As I have been using all of the new stuff I found myself adding my own custom functionality…. and after a while I realized (finally) that I should put all of those little helper things into a library, specifically an IoT Expert library.  For now this library contains:

      • The Bluetooth Platform Configuration Settings
      • Functions to decode stack events
      • Functions to decode advertising packets

      but imagine with time I will add more functions and templates.

      bt_platform_cfg_settings

      As I discussed in the article AnyCloud Bluetooth Advertising Scanner (Part 1), every single AnyCloud BLE Stack project will require a structure of type wiced_bt_cfg_settings_t.  This structure is used to setup the hardware before getting the stack going.  Remember you need to make a call like this to initialize the Bluetooth hardware.

      At some point this file will be included automatically for you by the Modus Toolbox team, but for now I have added this to my library.  This file look like this.  Basically a bunch of pin definitions which are set for you automatically by the BSP.  Obviously you can make your own, but this should work on all of the Cypress development kits (I think)

      btutil_stack

      When you startup the Bluetooth Host stack you are responsible for providing a “management callback” which has the function prototype

      The first parameter is an “event” which is  just an enumerated list of possible events by the Bluetooth Host Stack.  Here is the actual list.

      While you are trying to figure out what is going on during the development, it is very useful to be able to print out the name of the events (instead of the numbers).  In other words instead of doing

      it is way better to do

      While I was looking at an example project I found this function which I thought was awesome.

      And then I learned something new when I looked at the “function” CASE_RETURN_STR which used compiler trick to turn an enumerated value into a string.

      This allows you to do this in your management callback (notice line 16 where the string is printed out)

      It turns out that there are several other places where the stack gives you an event-ish thing.  So these functions were created as well

      btutil_adv_decode

      In AnyCloud Bluetooth Advertising Scanner (Part 3) I discussed the format of the advertising packet.  So I created functions which will decode the data in the advertising packets.  More on the future article AnyCloud Bluetooth Advertising Scanner (Part 4 or maybe 5 or maybe 6).  Here are the functions:

      btutil

      And because I like to have just one include to get access to all of the function I created “btutil.h” which just includes all of the headers in one place.

      Add to the IoT Expert Manifest

      In order to set it up for my library to live in the library manager I updated the IoT Expert Manifest file to have a link to the GitHub repository

      And now the library manager has the IoT Expert Bluetooth Utilities.

       

       

      AnyCloud Bluetooth Advertising Scanner (Part 3)

      Summary

      In this article I discuss BLE “Observing” and add that functionality to my PSoC 6 – CYW43xxx AnyCloud BLE Adverting Scanner project.

      Story

      In part1 of this series I discussed the pieces parts required to get the AnyCloud Bluetooth Stack operating using the AnyCloud SDK running on a PSoC 6 with a CY43xxx combo.  Then in part 2 I built a project with those parts and started up the Bluetooth Host stack.  The project didn’t really do anything, actually nothing, so it wasn’t very interesting, but it was going.  In this article I will discuss BLE advertising scanning, how to configure it in the AnyCloud project and finally how to add it to the project.

      For this series of articles I will:

      1. Discuss an overview of Bluetooth Application Architecture (Part 1)
      2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
        1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
        2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
        3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
        4. Build the Basic Project (Part 2)
        5. Explain BLE Advertising - Scanner/Observer (Part 3)
        6. Explain BLE Advertising - Advertiser (Part 3)
        7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
        8. Add Observing to our Project (Part 3)
        9. Program and Test the Project (Part 3)
        10. Add the IoT Expert Bluetooth Utility Library (Part 4)
        11. Mulithreading Dangers (Part 4)
        12. Add a Queue to the Bluetooth Manager Thread (Part 4)
        13. Add a Timer to Process the Queue (Part 4)
        14. A Potential Threading Bug (Part 4)
        15. Add a Scan Off & On Command (Part 4)
        16. Create an Advertising Data Database Task (Part 5)
        17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
        18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
        19. Create an Advertising Data Database (Part 5)
        20. Add a Command to Print the Database (Part 5)
        21. Refactor a silly duplication (Part 6)
        22. Update the print (Part 6)
        23. Add the "decode" command (Part 6)
        24. Add the decode packets functionality (Part 6)
        25. Program and test (Part 6)
        26. Design a new advertising database (Part 7)
        27. Update to the new database (Part 7)
        28. Update printing (Part 7)
        29. Update the advertising add (Part 7)
        30. Create a watch command (Part 7)
        31. Create a record command (Part 7)
        32. Create an erase command (Part 7)
        33. Update the filter data structure (Part 8)
        34. Fix the Add function (Part 8)
        35. Add the filter command (Part 8)
        36. Update the printing (Part 8)
        37. Fix a memory leak (Part 9)
        38. Add packet age (Part 9)
        39. Fix the printing, again (Part 9)
        40. Fix the add packet function (Part 9)
        41. Update the database structure for sort and purge (Part 10)
        42. Add the sort and purge commands (Part 10)
        43. Add the sort functionality (Part 10)
        44. Add the purge functionality (Part 10)
        45. The End & Commentary (Part 10)

        All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

        There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

        Explain BLE Advertising – Scanner/Observer

        You might recall that there are four roles that a BLE device can perform

        • Peripheral – low power devices that broadcast advertisements, then accept a single connection
        • Central – devices like cell phones that connect to peripherals.  They may run multiple connections at a time.
        • Broadcaster – a nonconnectable peripheral that sends out advertisements
        • Observer – A central-like device that listens for broadcasters (or advertising peripherals)

        And you might remember that advertisements are short, up to 31-byte, packets of data that give information which can be used for one or more of:

        • advertising the availability to connect
        • advertising services
        • advertising the name
        • advertising vendor specific information
        • advertising beacon data (like temperature or …)
        • advertising location

        And, if you forgot, BLE operates on 40 channels.  But to save power in peripherals, all of the advertising happens on channel 37, 38 and 39.  Specifically a peripheral or broadcaster will send out the advertising data on channel 37, then 38 then 39, then wait… then do it again.  But why one channel at a time?  Because BLE radio’s can be tuned to transmit and receive on only one channel at a time (a power saving and complexity reducing feature)

        Inside of the Central/Observer it will listen on channel 37 for a “window” amount of time.  Then it will do nothing for an interval-window amount of time.  Then it will do that same thing on channel 28 then 39.  But why only one channel at a time?  Same reason as above, it saves power and simplifies the design.  Why not have the window and the interval be the same?  Once again, it saves power.

        Here is a picture:

        But, what happens if you are not listening when the advertiser advertises?  You missed it.  Tough shit.  It turns out that setting the scan window and interval will greatly impact the probability that you hear advertisements.  And, you are more likely to hear advertisements because they are sent on three channels.  But it seems like it will never work.  Will it? … yes, of course, or they wouldn’t have done it that way 🙂

        BLE Advertising – The Advertiser Peripheral or Broadcaster

        So what exactly is inside of an advertising packet?  Volume 6 part B Section 2.3 of the bluetooth core spec describes the advertising protocol data unit (PDU)

        But what is inside of the header?

        This leaves us with what is inside of the “payload”.  The answer is that the ADV_IND Payload Data Unit (PDU) contains an address of 6-bytes plus up to 31 bytes of data.

        The AdvA field shall contain the advertiser’s public or random device address as indicated by TxAdd.

        The actual AdvData field is further broken up into “AD Structures” like this:

        And what is the “AD Type”, well it is a one byte of one of the following:

        And then where do you find the assigned numbers for the field types?  In the “Assigned Numbers and GAP“.  Here is a clip from the spec.

        And conveniently enough we enumerated them for you inside of the SDK header file wiced_bt_ble.h

        How does scanning work in the AnyCloud Bluetooth Stack?

        To turn on observing/scanning you need to call the function:

        Which will cause the host stack to tell the controller to start scanning for advertising packets.  It will set the scan window and scan interval the low duty scan settings from the bluetooth configuration structure… which we setup with the Bluetooth configurator.

        When the controller hears an advertising packet, it will send the HCI advertising report to the Bluetooth host stack, which with then call you back.  Specifically it will call you back by calling the p_scan_result_cback” function.

        You provide the callback function which has the prototype:

        which contains two parameters, p_scan_result which is a structure that has the mac address and some thing data plus the p_adv_data which has the raw bytes of the advertising packet.

        Add Observing to our Project

        OK.  Lets add this to our project by creating a callback function like this:

        Lines 5-9: Just prints out the raw bytes of the MAC address of the remote device, the one advertising

        To print out the raw advertising data you need to remember that it is formatted as

        1. A length (of all of the data of the field)
        2. A type
        3. The rest of the data

        When you find a field of length of 0 you know that you have reached the end of the data

        On Lines 13-20: I print out one field at a time and the raw data

        Then update the management callback to start the scanner after the stack is successfully started

        Program and Test

        Now when I run the program data comes blasting out of the screen because there are a boatload of ble devices in my house

        In the next article Ill add some more smarts to manage the data to be easier to look at.

        For your information here is all of the file bluetoothManager.c

         

        AnyCloud Bluetooth Advertising Scanner (Part 2)

        Summary

        The second article in a series discussing the creation of a PSoC 6 + CYW43xxx Advertising Scanner using the AnyCloud SDK.  This article will use the learning from Part 1 to create a template project that starts the BLE stack.

        Story

        In the previous article I discussed the structure of the Cypress/Infineon Bluetooth Stack and its integration into AnyCloud.  A bunch of “theory”, well I say BS to that.  Let’s build something.

        For this series of articles I will:

        1. Discuss an overview of Bluetooth Application Architecture (Part 1)
        2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
          1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
          2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
          3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
          4. Build the Basic Project (Part 2)
          5. Explain BLE Advertising - Scanner/Observer (Part 3)
          6. Explain BLE Advertising - Advertiser (Part 3)
          7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
          8. Add Observing to our Project (Part 3)
          9. Program and Test the Project (Part 3)
          10. Add the IoT Expert Bluetooth Utility Library (Part 4)
          11. Mulithreading Dangers (Part 4)
          12. Add a Queue to the Bluetooth Manager Thread (Part 4)
          13. Add a Timer to Process the Queue (Part 4)
          14. A Potential Threading Bug (Part 4)
          15. Add a Scan Off & On Command (Part 4)
          16. Create an Advertising Data Database Task (Part 5)
          17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
          18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
          19. Create an Advertising Data Database (Part 5)
          20. Add a Command to Print the Database (Part 5)
          21. Refactor a silly duplication (Part 6)
          22. Update the print (Part 6)
          23. Add the "decode" command (Part 6)
          24. Add the decode packets functionality (Part 6)
          25. Program and test (Part 6)
          26. Design a new advertising database (Part 7)
          27. Update to the new database (Part 7)
          28. Update printing (Part 7)
          29. Update the advertising add (Part 7)
          30. Create a watch command (Part 7)
          31. Create a record command (Part 7)
          32. Create an erase command (Part 7)
          33. Update the filter data structure (Part 8)
          34. Fix the Add function (Part 8)
          35. Add the filter command (Part 8)
          36. Update the printing (Part 8)
          37. Fix a memory leak (Part 9)
          38. Add packet age (Part 9)
          39. Fix the printing, again (Part 9)
          40. Fix the add packet function (Part 9)
          41. Update the database structure for sort and purge (Part 10)
          42. Add the sort and purge commands (Part 10)
          43. Add the sort functionality (Part 10)
          44. Add the purge functionality (Part 10)
          45. The End & Commentary (Part 10)

          All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

          There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

          Recall from Part 1 that you need three things to startup the Bluetooth Stack

          • The Hardware Configuration Structure that matches : cybt_platform_config_t
          • The Bluetooth Stack Configuration Structure that matches : wiced_bt_cfg_settings_t
          • The Bluetooth Management Callback that matches : typedef wiced_result_t (wiced_bt_management_cback_t) (wiced_bt_management_evt_t event, wiced_bt_management_evt_data_t *p_event_data);

          Then you need to call

          • The hardware initialization function : cybt_platform_config_init
          • The stack initialization function : wiced_bt_stack_init

          Ok let’s do this!

          Basic Project

          You can do all of these steps from the Eclipse IDE for ModusToolbox.  Or you can do it from the individual programs and the command line.  I like Visual Studio code, so this article will be done completely from the command line and individual configurators.

          Run the new project creator from the start menu.  Start by creating a project for the development kit that you have, in my case the one currently plugged into my computer is the CY8CKIT-062S2-43012, so that is what I pick.  But, this project will work with any of the WiFI/BT combo chips attached to PSoC 6.

          In previous articles I discussed the template that I use to get things going with FreeRTOS.  I won’t discuss that here, but I want FreeRTOS and the NTShell, so pick the IoT Expert FreeRTOS NTShell Template.

          After about a minute you should have a project.  I always like to build the project to make sure that everything is working before I get too far down the road of modifying anything.  Run “make -j build”

          Then to be sure it is working, program the development kit.

          When that is done, open up a terminal window and you should have a functioning base project.  Notice that I ran “help” and “tasks” from the command shell.

          Now that we have a basic project working, add the Bluetooth libraries.  Run the library manager by typing “make modlibs”.  Then select “bluetooth-freertos” and the library manager will automatically select the other libraries you need.  Press Update then Close.

          Next, run the bluetooth configurator by running “make config_bt”  This tool will help you make the bluetooth stack configuration structure.  When the configurator starts, press “New”

          Then select our device (the PSoC 6 and the Combo chip)

          Click on the “GAP Settings”.  Then press the Plus and add “Observer configuration”

          Then setup the scan settings (more detail on these numbers in the next article)

          • Low duty scan window (ms) = 60
          • Low duty scan interval (ms) = 60
          • Low duty scan timeout = deselected (meaning no timeout)

          Then save your configuration file.  Notice that I called it “btconfig”

          When you are done you will have a directory called “GeneratedSource” inside of your project with the needed files.

          The next step is to fix up the Makefile.  I like changing the name of the “App”.

          Then you need the “FREERTOS WICED_BLE” components.

          If you run make vscode it will update the workspace with all of the stuff needed for Visual Studio Code to be able to find all of the files.

          Inside of Visual Studio Code, create a new file called “bt_platform_cfg_settings.h” and add:

          Inside of Visual Studio Code, create a new file called “bt_platform_cfg_settings.c” and add:

          Inside of Visual Studio Code, create bluetoothManager.h.  Remember this is the Bluetooth Stack Management Callback

          Inside of Visual Studio code, create bluetoothManager.c.  This function does a whole lotta nothin… except saying that things got started.

          Next, update main.c with the required includes.

          Then update main.c to start the stack

          Now build and program it, remember “make -j build” and “make program”.  Look, we have a functioning stack with the two bluetooth thread running.

          In the next article Ill finally get around to building the Bluetooth Scanner.

          AnyCloud Bluetooth Advertising Scanner (Part 1)

          Summary

          The first of several articles discussing the use of the AnyCloud BLE Stack to build advertising scanner/observers.

          Story

          A few summers ago while I was writing the WICED Bluetooth Academy book, I created a WICED based BLE advertising scanner.  Actually, I created a framework for the scanner and the remarkable summer intern I had finished the work.  That project has been sitting in the source code repository for the Bluetooth class, mostly only shown to face-to-face students.  This scanner is built using some of the original code combined with the new AnyCloud Bluetooth SDK.  It will act sort-of-like LightBlue or one of the other Bluetooth advertising scanners you might run on your phone, but with a serial console.

          Sometime in the last few months we released the Bluetooth SDK for AnyCloud (things have been crazy and I have lost track of time)  This SDK has all of the stuff needed to add Bluetooth to your AnyCloud project using one of the Cypress Bluetooth/WiFi combo chips.  I had not had a chance to try it out, so I decided to build a Bluetooth project and then port the scanning code.

          For this series of articles I will:

          1. Discuss an overview of Bluetooth Application Architecture (Part 1)
          2. Explain how to Start the AnyCloud Bluetooth Stack (Part 1) 
            1. Discuss the configuration structure: bt_platform_cfg_setting (Part 1)
            2. Discuss the configuration structure: wiced_bt_cfg_settings (Part 1)
            3. Discuss the Tasks created by the Bluetooth Stack (Part 1)
            4. Build the Basic Project (Part 2)
            5. Explain BLE Advertising - Scanner/Observer (Part 3)
            6. Explain BLE Advertising - Advertiser (Part 3)
            7. How does scanning work in the AnyCloud Bluetooth Stack? (Part 3)
            8. Add Observing to our Project (Part 3)
            9. Program and Test the Project (Part 3)
            10. Add the IoT Expert Bluetooth Utility Library (Part 4)
            11. Mulithreading Dangers (Part 4)
            12. Add a Queue to the Bluetooth Manager Thread (Part 4)
            13. Add a Timer to Process the Queue (Part 4)
            14. A Potential Threading Bug (Part 4)
            15. Add a Scan Off & On Command (Part 4)
            16. Create an Advertising Data Database Task (Part 5)
            17. Update the Advertising Database to Accept Submitted ADV Packets (Part 5)
            18. Update the Bluetooth Manager to Submit Adv Packets (Part 5)
            19. Create an Advertising Data Database (Part 5)
            20. Add a Command to Print the Database (Part 5)
            21. Refactor a silly duplication (Part 6)
            22. Update the print (Part 6)
            23. Add the "decode" command (Part 6)
            24. Add the decode packets functionality (Part 6)
            25. Program and test (Part 6)
            26. Design a new advertising database (Part 7)
            27. Update to the new database (Part 7)
            28. Update printing (Part 7)
            29. Update the advertising add (Part 7)
            30. Create a watch command (Part 7)
            31. Create a record command (Part 7)
            32. Create an erase command (Part 7)
            33. Update the filter data structure (Part 8)
            34. Fix the Add function (Part 8)
            35. Add the filter command (Part 8)
            36. Update the printing (Part 8)
            37. Fix a memory leak (Part 9)
            38. Add packet age (Part 9)
            39. Fix the printing, again (Part 9)
            40. Fix the add packet function (Part 9)
            41. Update the database structure for sort and purge (Part 10)
            42. Add the sort and purge commands (Part 10)
            43. Add the sort functionality (Part 10)
            44. Add the purge functionality (Part 10)
            45. The End & Commentary (Part 10)

            All of the code can be found at git@github.com:iotexpert/AnyCloudBLEScanner.git and https://github.com/iotexpert/AnyCloudBLEScanner.git

            There are git tags in place starting at part 5 so that you can look at just that version of the code.  "git tag" to list the tags.  And "git checkout part6" to look at the part 6 version of the code.

            Bluetooth Application Architecture

            Bluetooth applications are divided into these four pieces

            1. You user application which responds to events and sends messages from/to the Bluetooth host stack
            2. A Bluetooth Host Stack
            3. A Bluetooth Controller Stack
            4. The Bluetooth Radio

            These four pieces can be divided into multiple chips, as few as one or as many as four.  However, for this article, the project will be built to run on a PSoC 6 + CYW43012 WiFi/Bluetooth Combo chip.  Specifically:

            1. My scanner application running on the PSoC 6
            2. The Bluetooth Host Stack running on the PSoC 6
            3. The BlueTooth Controller Firmware running on the CYW43012
            4. A Bluetooth Radio on the CYW43012

            But how do they talk?  Simple, there is:

            1. A UART Host Controller Interface (HCI) between the two chips
            2. A GPIO to serve as a deep sleep wakeup from the 43012 –> PSoC 6
            3. A GPIO to serve as the bluetooth controller wakeup from the PSoC 6 –> 43012
            4. A GPIO to turn on the Bluetooth regulator from the PSoC 6 –> 43012

            Here is the block diagram from the CY8CKIT-062S2-43012 Kit Guide.  Notice that signals labeled UART and Control going between the PSoC 6 and the CYW43012.

            And when you read more deeply into the schematic you can see the signals labeled

            • BT_UART_TXD/RXD/CTS/RTS
            • BT_HOST_WAKE
            • BT_DEV_WAKE
            • BT_REG_ON

            How to Start the AnyCloud Bluetooth Stack

            To actually start the AnyCloud Bluetooth stack you will call two functions

            1. cybt_platform_config_init – that will setup the hardware interface to the CYW43012
            2. wiced_bt_stack_init that will:
              1. Start a task to manage the Host Controller Interface
              2. Download the controller firmware to the CYW43012
              3. Start a task to manage the host stack
              4. Initialize both the host and the controller
              5. Call you back when that is all done

            Here is an example from main.

            When you look at these two function calls, you will find that you need to provide three things:

            1. A platform hardware configuration structure called bt_platform_cfg_settings
            2. The Bluetooth stack configuration settings structure called wiced_bt_cfg_settings
            3. A management callback called app_bt_management_callback

            bt_platform_cfg_settings

            The purpose of the hardware configuration structure is to define the UART + parameters and the wakeup GPIOs.  Specifically, the hardware configuration structure defines the configuration of the host controller interface (hci)

            1. The HCI transport scheme (in this case UART)
            2. The pins of the UART
            3. Baud Rate
            4. Data Bits
            5. Stop Bits
            6. Parity
            7. Flow Control

            And the controller low power behavior (in the .controller_config member)

            This is a fairly standard configuration and I think that we should help you by providing this structure somewhere in the BSP.  But for now, you need to provide it (in an upcoming article I’ll update the IoT Expert Bluetooth Library to provide it).  Here is the specific structure that I will be using.

            wiced_bt_cfg_settings

            The Cypress WICED Bluetooth Stack has a boatload of configuration settings.  When you call the stack start function you need to provide all of those settings in a structure of type “wiced_bt_cfg_settings_t” which is actually a structure of structures.  There are several basic ways to set these settings

            • Start from scratch and build you own settings
            • Copy from an example project
            • Use the Bluetooth Configurator to generate the structure

            For the purposes of THIS project I started by copying the structure from on of the example projects and then modifying the three numbers that were relevant to me.  Specifically

            • max_simultanous_link – which I changed to 0 because this is simply a Bluetooth Observer
            • low_duty_scan_interval – how long in the window to listen for advertising packets
            • low_duty_scan_window – how wide the window of listening should be

            app_bt_management_callback

            The last thing that you need to provide is a management callback.  This function is called by the Bluetooth Stack when a “management event” occurs.  There is a big-long-list of enumerated events of type wiced_bt_management_event_t.  The events include things like the the stack started “BTM_ENABLED_EVENT”.  Each event may have data associated with the event which is passed to you in a pointer to wiced_bt_management_event_data_t.

            You typically deal with these events with a giant switch statement like this:

            Tasks

            The Bluetooth stack on the PSoC6 is operated with two tasks.  Specifically, when you call the wiced_bt_stack_init it will startup:

            1. CYBT_HCI_Task – a task that sends and receives HCI packets going to the Radio chip
            2. CY_BT_Task – a task that manages the Bluetooth Host Stack

            Here is print of the task list from my project:

             

            Now with the background in place, in the next article I will discuss Bluetooth advertising and how to build the observer project.

            PSoC 6 BLE Events

            Edit: It turns out that I wrote this article in February of 2018 and never published it.  But it has good stuff… so here it is.

            Summary

            Over the last several weeks I have been spending time working with the PSoC 6 BLE.  Specifically the BLE part of the system.  Last weekend I found myself having problems with my design and I was really struggling to figure them out.  I realized that I “knew” what was happening with the BLE callbacks… but that I didn’t really “KNOW!”.  So I decided to build a simple project to show what events were happening and why.

            In this article I will show/tell you how (to):

            1. The BLE is supposed to work
            2. The PSoC 6 BLE example project
            3. Write the firmware
            4. Test the system

            You can “git” this workspace at git@github.com:iotexpert/PSoC-6-BLE-Events.git or GitHub.

            How the PSoC 6 BLE is supposed to work

            Honestly, the first time I started looking at the BLE (in my case it was the PSoC 4 BLE), it felt overwhelming.  But, it is actually pretty simple.  You, the firmware developer, send commands to the BLE stack using the Cypress PDL BLE middleware API.  Then, when “something” happens, the BLE stack sends you back an “event” in the callback.  That “something” can either be an acknowledgement that your API call has done something, or, it can be that something has happened on the radio side (e.g. you have been connected to by a client).  When you write a PSoC 6 BLE program you need to provide an event handler function (so that the BLE stack can send you events).  The function prototype for that handler is:

            This function has two arguments.

            1. An integer that is an eventCode (which you can put into the switch).  All of the event codes will look like “CYBLE_EVT_”
            2. A void pointer, that you will cast into a specific pointer based on the event code.

            Your event handler code will then be a big switch statement, where you will look at the events, and do something (or not).

            When you look in the PDL BLE Middleware documentation you can see the APIs and what events happen based on your API calls.  For example Cy_BLE_GAPP_StartAdvertisement tell the PSoC BLE Stack to start advertising.  You can see that it will generate 4 possible events i.e. CY_BLE_EVT_GAPP_START_STOP

            When you click on the event in the documentation it will tell you the meaning of the event, and what the eventParameter means (i.e. what you should cast it to in order to figure out the data passed to you)

            Build the project

            To build the project, first make a new PSoC 63 BLE project.  Then edit the schematic to have a BLE and a UART.

            PSoC 6 BLE Events Schematic

            Assign the UART pins to the KitProg UART bridge pins (aka P50/P51)

            PSoC 6 BLE Events Pin Assignment

            Configure the BLE to be a GAP Peripheral.

            PSoC 6 BLE Events Configuration

            Add a custom service to the project by loading the LED Service.  It doesn’t really matter what service you add for this project.  I just picked this one because I am using it for another project.  You could have just as easily picked one of the pre-existing definitions or made your own.

            PSoC 6 BLE Events Configuration

            This is what the LED Service looks like.

            PSoC 6 BLE Events - LED Service

            Configure the GAP settings.  Specifically name your device – in this case I named mine “mytest”

            PSoC 6 BLE Events GAP Configuration

            Edit the advertising packet to include the name and the service.

            PSoC 6 BLE Events Advertising Configuration

            Write the Firmware

            Remember the main event in this example project is the BLE Event Handler.  I created this event handler with the events that I normally used (i.e. CY_BLE_EVT_STACK_ON) and then kept adding events until I had them all defined.  The way that I knew an event was missing from the “switch” was by the default case printing out the event number.

            Now you need a task to run the BLE.  It just runs the Cy_BLE_ProcessEvents each time an event needs to be handled.

            Test the System

            Finally program the CY8CKIT-062-BLE and attach with it to CySmart or LightBlue.  There are three phases

            1. The stack turns on and starts advertising (CY_BLE_EVT_STACK_ON –> CY_BLE_EVT_GAPP_ADVERTISMENT)
            2. A connection is made and eventually disconnected (CY_BLE_EVT_CONNECT_IND –> CY_BLE_EVT_GAP_DEVICE_DISCONNECTED
            3. You start advertising again CY_BLE_EVT_GAPP_ADVERTISEMENT_START_STOP

            PSoC 6 BLE Events

             

            MBEDOS & BLE & PSoC 6 & CYW4343W

            Summary

            At the Embedded World Show in Germany in a couple of weeks I am going to be showing a crazy demo (more on this later) that uses MBED OS and BLE and WiFi and PSoC 6 and the 4343W.  Given how close things are and how new MBED OS is to me I figure that I had better get going sorting out the BLE interface.   This article and probably the next several are going to show my progress through the learning curve.

            It turns out that in MBED OS, instead of using the Cypress BLE Host Stack I will be using the ARM Cordio BLE host stack talking via HCI to the Cypress BLE Controller stack running on the 4343W (a Bluetooth, BLE and WiFi combo chip).  At this point all of my experience with BLE has been with Cypress stacks, either the PSoC 4/6 BLE stack or with the Cypress IoT stacks e.g. the CYW20719.  Lot’s of new learning.  Add in that all of the code is in C++ and it makes for an adventure.

            For this article I will show the steps to get an ARM BLE example going on the CY8CPROTO_062_4343W development kit.  This will involve.

            1. Importing the ARM MBEDOS BLE Examples
            2. Modifying them to support the Cypress Targets & Test
            3. Updating an example program in a few places to fix things that I don’t like.

            Import ARM MBED OS BLE Examples

            The first step is to make a clone of the ARM examples by running “mbed import mbed-os-example-ble”.  This will load a bunch of different example projects (as libraries)

            Then, when you look at what you have after all of that mess, you can see 14 example programs with promising names.

            When you look in the BLE_LED directory you will find a file “readme.md” which is a markdown formatted file.  You can view this file on the GitHub website for this example here.  The top of this one looks promising:

            Modify and Test

            I decide that the example called “BLE_LED” looks like a good place to start.  This example is a simple peripheral that advertises it name.  When you connect to it there is a Service with UUID “0xA000” (unfortunately a 16-bit UUID… bad demo code) that Service has one characteristic with UUID 0xA001 (another 16-UUID … that isn’t nice … come on people… haven’t you read the spec?).  When you write a “1” to that characteristic the LED2 is supposed to turn on, and when you write a 0 the LED2 is supposed to turn off.

            First, until the Cypress stuff is accepted into the main release, I need to update mbed-os to our targets) with “cd mbed-os ; mbed update master”.  To build this project Ill run “mbed compile -t GCC_ARM -m CY8CPROTO_062_4343W”.  When I program the the development kit, the LED starts blinking and I am able to see it using the GATT browser LightBlue Explorer.

            But when I try to write a 1 to the 0xA001 characteristic nothing happens.

            So, what gives? The answer is that on line 32 you can see that the authors as assuming that you have two LEDs (my development kit only has one.

            And on line 124 you can see a function that inverts the LED

            which is triggered on line 47 to be called every 500ms

            OK.  I am not loving this. I think that I should make some updates to this project.

            Update

            There are several things that I don’t like about this program or need to be fixed.

            1. Make the user LED2 be LED1 and fix the fact that it is active low.
            2. Change the UUIDs of the Service and Characteristic to be legal 128-bit UUIDs
            3. Make the stdio print out the status of the connection (instead of the blinking LED1)
            4. Make the baud rate of standard i/o be 115200 instead of 9600

            First, fix the LED2 to be LED1.  Do this by commenting out all of the _alive_led code and switching the _actuated_led to be LED1.  Also set the state of the LED1 to 1 (meaning off because it is active low)

            The author of the example code has a function called blink which is executed by the event queue every 500ms, comment out that function

            And don’t inject events into the queue to run the blink function

            The LED on my board is active low… so instead of writing the value write the opposite of the value.

            It is illegal in Bluetooth to use 16-bit UUIDs without first registering them with the Bluetooth SIG and having them be “Assigned numbers”.  The author of this example program violated the specification by assigning the LED service UUID of 0xA000 and the LED characteristic UUID of 0xA001.  This is super annoying and I am not willing to be a slob.  To fix this modify ledservice.h to declare the UUIDs as UUID type instead of uint16_ts

            Then initialize them in the main.cpp as 128-bit UUIDs using the const char * initializer.

            The original code has a blinking LED.  Which I dont really like.  Typically, I like to blink the LED when the device is advertising, and make it be solid when there is a connection.  However, as I only have one LED on my board, and I have allocated it to be the “_actuated_led”, I will use the UART to print out status changes.  To do this, I update the “onDisconnectionComplete” and “onConnectionComplete” events to print out that fact to stdio.

            In order to set the stdio to use 115200 instead of 9600 you can change the default rate of the UART in the mbed_app.json.

            Here is the final version of main.cpp