Tag Archives: OTA

Using mklink to centralize Teensy OTA Files

Posted 20 December 2022

This post describes the actions I have taken to centralize the ‘board.txt’ and ‘TeensyOTA1.ttl’ (Tera Term macro) locations that facilitate the Teensy OTA capability, so that all my various firmware projects targeted at my wall-following robot all use the same set of files. This is done by using the Windows ‘mklink’ command to create ‘hard’ links from the various program folders to a single folder called ‘Robot Common Files’.

For the past five or six years I’ve been working on a 4-wheel robot project to autonomously navigate around my house using a (by now) fairly sophisticated wall-following algorithm. As I have worked through the various problems, I’ve gone through at least three different physical form factors, starting with a 3-wheel (two driven wheels, one castering wheel) and ending up (so far) with a custom 4-wheel ‘wide-track’ model.

Also along the way I have gone through any number of software versions. I work mainly with the Arduino ecosystem, but I don’t use Arduino directly. I use Microsoft Visual Studio 2022 Community Edition with the Visual Micro extension, and this is a great platform.

My latest hardware upgrade was to replace my original Arduino Mega 2560 processor with a Teensy 3.5 main processor, with a firmware update to achieve over-the-air (OTA) updates using a Wixel RF pair.

I now have at least twenty different software/firmware projects that I run on the robot for different reasons. I have a ‘main line’ project that incorporates all features required to full execute autonomous wall-following, and then I have lots of smaller projects aimed at exercising some small subset of the full feature set – things like distance calibration testing for the robot’s two 3-element VL53L0X time-of-flight distance sensors, and for testing out different ways of managing ‘rolling’ and ‘spin’ turns, and different algorithms for wall-tracking. All of these projects use the same ‘over-the-air’ (OTA) firmware/hardware configuration for firmware updates. Up until now I have been simply copying the two required files – ‘board.txt’ and ‘TeensyOTA1.ttl’ (a Tera Term macro) from project folder to project folder, but a recent Visual Micro update broke my OTA routine, and I discovered that having these two files copied into multiple project folders didn’t work so well – oops!

So, I decided to create ‘Robot Common Files’ folder in my Documents\Arduino folder tree, place these two files (along with the required FlashTxx.cpp/h files and my ‘enums.h’ file) into this folder, and then use the ‘mklink /H’ command (12/23/22 note: Admin privileges are not required) to create ‘hard’ links from each project’s folder to the single files in m Robot Common Files folder. The commands I used to accomplish this were:

where ‘WallE3_AnomalyRecovery_V2’ gets replaced each time with the actual project name

Now, when I want to edit either the ‘board.txt’ or ‘TeensyOTA1.ttl’ file, I can simply right-click on the filename in any project folder and select ‘edit with NotePad++’, and the single file in Robot Common Files gets opened for edit, and the changes are immediately available in all projects that use the Teensy OTA feature.

31 January 2023 Update:

After trying this trick a few times, I realized I had left out a step, so this update fixes that. Below is the entire Cmd line session for my latest ‘new project’:

The first line above shows the ‘cd’ step, which I had left out of the previous post. After that, I just copy/pasted the ‘mklink….’ text for each of the two files, and then everything was wonderful again.

Stay Tuned,

Frank

Over-the-Air (OTA) Firmware Updates For Teensy 3 & 4.x, Part II

Posted 16 October 2021,

In my previous post on this subject, I described my effort to automate over-the-air (OTA) updates of a Teensy 3.5 using a C# command-line program called by a post-build script from Visual Studio 2019 with the Visual Micro Arduino IDE extension, combined with Joe Pasquariello’s find ‘FlasherX’ code. As the article described, I was successful in doing this, but the time required to transfer the .HEX file using the C# .NET serial interface appeared to be about 2-3 times slower than that required by using the Tera Term serial comms program manually.

So, I went back to the drawing board, and started searching for a way to automate Tera Term, rather than building my own serial port management applet. I found the Tera Term help index, and subsequently I found that there is indeed a quite rich macro language associated with Tera Term, so I started learning the language and trying to apply it to my problem of using Visual Micro’s post-build commands via the ‘board.txt’ feature to automate the process of OTA for the Teensy microcontroller.

After the normal amount of fumbling around with the macro language, I was able to create a reasonably functional macro file that accepts three arguments from the Visual Micro ‘board.txt’ post-build command line and then automates the process of uploading the associated .HEX file to my Teensy 3.5 and then rebooting the Teensy to run the updated program. The ‘board.txt’ contents are:

The post-build command launches the Tera Term Pro Macro interpreter (ttpmacro.exe) and passes three arguments; the first is the build path – the folder in which the compiled .HEX file will be placed. The second is the COM port number assigned to the HC-05 bluetooth link with my laptop, and the third is the project file name, i.e. ‘TeensyOTADemo.INO’.

The Tera Term macro uses the first and third argument to build a path to the compiler’s .HEX file for the project, i.e. <build_path>\<project file name minus extension>.HEX. Then it uses the COM port specified in the second argument to connect to the Teensy and send a ‘trigger character’ to force the Teensy into ‘update mode’, and upload the .HEX file. After the .HEX file has been uploaded, the macro responds to the ‘enter xxxx to update or 0 to abort’ response from the Teensy by sending back the xxxx value, which causes the Teensy to reboot and begin running the new program.

Here is the complete Tera Term macro file:

and here are two short videos showing the OTA update process. The first video shows the process from the PC’s point of view, and the second one shows the same thing, but from the Teensy’s point of view.

Teensy OTA from PC’s point of view
Teensy OTA from Teensy’s point of view

Summary:

Over-the-air (OTA) update of a Teensy microcontroller is now practical using Joe Pasquariello’s fine ‘FlasherX’ program, combined with a Tera Term macro that is launched using Visual Micro’s ‘board.txt’ post-build feature. The things you need to make this happen are:

  • Obviously any Teensy program must incorporate Joe Pasquariello’s ‘FlasherX’ functionality. See my previous post for my complete demo sketch that does this.
  • You must have a way of triggering the update functionality provided by ‘FlasherX’. In my demo I accomplished this via my ‘CheckForUserInput()’ function that runs each time ‘loop()’ executes. If this function detects the letter ‘U’ or ‘u’ on Serial1, it launches FlasherX’s ‘update_firmware’ function which does the rest.
  • You have to have a serial comms application to upload the .HEX file produced by compiling the program update. I used Tera Term for this, and it worked very well.
  • To automate the above process, you need a script file (macro) like the one I provided above to manage the upload process.

I created a new GitHub repository here containing the Tera Term macro file, the ‘board.txt’ file I used with VS/VM, and the OTA demo sketch I used to demonstrate this functionality. Enjoy!

Frank

27 October 2021 Update:

As part of my Wall-E3 project, I constructed a small perf-board module to carry the low-dropout (LD0) 5V Regulator, and added a HC-05 module for OTA updates to Wall-E3’s Teensy firmware, as shown in the following photo

5V regulator board with HC-05 BT module added for OTA updates

And here is a short video showing an OTA update:

Teensy OTA firmware update using HC-05 BT module and Tera Term script

Over-the-Air (OTA) Firmware Updates For Teensy 3.x & 4.x

Posted 23 September 2021

For some time now I have been following developments on Paul Stoffregen’s Teensy forum with respect to OTA updates for Teensy products. As it turns out, doing OTA updates is less than straightforward due to the Teensy architecture (I’m waving my hands here).

However, in recent months, thanks to Teensy forum member joepasquariello, there is a new set of programs that seem to address the Teensy OTA issue in a much more comprehensive manner. See this thread (starting with post #85) on the Teensy forum for the details. After reading through the posts, and asking some more dumb questions (which joe was fortunately willing to answer), I decided to give this a try to see if I could achieve OTA updates for a Teensy controller, and maybe make it as convenient as OTA updates are now with Arduino products using the Pololu Wixel RF serial pipe.

The FlasherX package is actually three different files; FlasherX.ino is a demo program that #includes FlashTxx.h and FlashTxx.cpp. The FlasherX.ino program is loaded into the target Teensy device in the normal manner, just like any other Teensy program. However, once it is there, it adds the capability to update the Teensy’s firmware using the .HEX file (produced by the normal compile process) transmitted over any serial port. Of course, in order to make this magic happen, the updated Teensy program must also have this capability, so that means that any Teensy program destined for OTA updates must include the basic ‘FlasherX’ components.

In order to get started, I dl’d all three files and programmed a Teensy 3.5 with the FlasherX.ino program using my normal VS2019/Visual Micro setup. Then I created a new Arduino Project in VS2019 called ‘TeensyFlasher_V1’ and copied the entire contents of FlasherX.ino. I compiled it for a T3.5 target and used my normal Teensy bootload process to transfer it to the Teensy 3.5. Here is the serial port output after the upload:

Initial output from FlasherX.ino after upload to Teensy 3.5

Next I used Tera Term’s ‘file send’ capability to send the FlasherX.hex file to the Teensy using the wired USB connection as just another serial port. When I first launched Tera Term I got an error dialog saying it “Cannot open COM11. Access denied” – reminding me that I had to close the normal connection in order for Tera Term to operate properly. Once I had done this, I used the ‘File->New Connection menu item in Tera Term to create a new connection to COM11 (the Teensy 3.5 USB port), as shown:

New connection to Teensy port

Next I used ‘File->Send File… to select the .HEX file created by the compiler, and transfer its contents over to the Teensy using the COM11 serial port

After selecting the .HEX file and clicking ‘Open’, the HEX contents were sent over the serial connection as shown below:

The reason for using the HEX file created from compiling FlasherX.ino (as opposed to a random Teensy app) is because FlasherX.ino already contains all the pieces needed to do OTA updates. Although a bit recursive, using the FlasherX app demonstrates that a Teensy device can be updated via a serial connection repeatedly, as long as the newly updated firmware contains the pieces needed to support the next OTA update.

The next step was to figure out how to get VS2019/Visual Micro to do the update automatically, so that the process emulates what happens with an Arduino controller when F5 is pressed. VS2019 supports post-build scripts, but they aren’t used in the Visual Micro environment. Instead, as I learned from VM guru Tim Leek, VM supports post-build operations via a special file called ‘board.txt’ in the project directory. If this file is present, properly formatted lines are treated as ‘build property overrides’. For instance, this is the line in ‘board.txt’ that calls a C# console app to actually communicate with the target Teensy.

This command launches the console app, and provides it with the path to the project, the path to the sketch, and the currently selected port number. See this thread on the VM forum for all the gory details.

I ran into a problem when I tried to use the above command line with my C# console app, because the arg[] string array in the console app was interpreting spaces in the strings in the calling command differently than expected, so when I sent three strings to the command, the console app thought I was only sending two – somehow concatenating the com port number into the second string (the sketch path). After playing around some more, I found that the following treatment:

when passed as an argument to this small code snippet

produced the following output:

With the correct number of arguments, and the comport number on its own line – yay. I have no idea why this works and the other formulations don’t, and why the second argument has to be have it’s quotes escaped with ‘\’, but hey – it works.

As it turns out, this exercise was a bit academic anyway, because both the {ProjectDir} and {sketch_path} expand to identical results, so I only need one of them and the comport number to proceed. So, the ‘final’ (for now) call argument list looks like:

which produces:

27 September 2021 Update

In order to do OTA updates, I need to be able to connect to one of the Teensy’s hardware serial ports. Up til now I have been just using the USB serial port for convenience, so I thought I would try using a FTDI Pro USB/Serial converter module to connect to Serial1 on the teensy, as shown in the following photo:

In the above photo, the normal USB connection provides 5V power for the Teensy, and also allows ‘direct’ programming via the normal VS2019/VM route. The FTDI module connection shows up as COM3, a ‘regular’ USB serial port. With this setup I was able to upload a .HEX file to the Teensy using Tera Term, confirming that I can indeed use a hardware serial port for this purpose.

04 October Update:

Between the last update and now, I spent a lot of time trying to get the command-line post-build console app working, mostly to no avail. I kept getting wrapped around the axle trying to deal with three different serial connections; two to the teensy itself, and one from the command-line app to VS2019’s Console service. This got to the point where at one time the ‘Console’ output was going to my computer display, which made the mouse jump around wildly, occasionally launching or closing apps! Even trying to restart the PC was a trial-and-error process, because about half the time the screen focus would jump away from the target button before I could get it clicked!

So, I decided to go back to basics and use just the FTDI serial adaptor to run the Teensy. This required using the Vcc connection from the FTDI module to power the Teensy, which worked OK once I figured out how to switch the FTDI module from 3.3V out to 5V out.

FTDI USB-serial module now powering Teensy 3.5

With this setup I am able to upload programs to the Teensy using Tera Term, as before, but now I don’t have the other USB connection to confuse things. It does mean however, that now I MUST use Tera Term to upload sketches, as I no longer have the capability to use the Teensy bootloader from VS2019.

With this simpler setup, ‘Console’ statements from the command-line app should go only to the VS2019 debug session, and serial port operations from the command-line app should go only to Teensy’s ‘serial1’ hardware serial port.

05 October 2021 Update:

I expanded the above operation to restore the USB connection to the PC, and created a small Arduino program to (at first) just echo serial data from one serial port to the other. Here’s the program:

Once I had this going to my satisfaction, I started adding pieces from my ‘TeensyFlashDemoV2’ project until I had a complete working program, as shown below:

Here’s the Arduino code:

and here is the companion C# command-line app:

And here is the command-line output from my last run:

In the above printout, most of the HEX lines were removed just to keep the line count down, but all 2977 lines were successfully sent to the Teensy. The lines

were emitted from the Teensy at the conclusion of the HEX file upload. The line

was from the command-line app comparing the number of lines sent with the number of lines received. The numbers matched, so ‘2977’ was sent to the Teensy to complete the update.

At this point I have a working automated update system except the COM port number and HEX file location was passed to the C# program as a set of fixed arguments (as shown below), to disconnect the command-line app from the Arduino sketch, instead of dynamically providing the same parameters from VM build properties

Now that I have some confidence that the command-line app does what it is supposed to do, I can go back and make that connection again, hopefully with better results this time.

A problem with the Arduino-to-C# command-line app is that the ‘build properties’ exposed by Visual Micro aren’t always the same, so what works for one sketch may or may not work for another. Here’s the ‘build properties’ output from my expanded ‘TeensySerialBridgeV1.ino’ file:

I need two things from the build properties – the COM port number, and the path to the .HEX file. From the above list, it appears that I could use either of the following properties,

but neither is very convenient, as the COM port number has to be extracted from a ‘known’ string. On other projects, I have seen a build property like ‘serial.port.number = 11’ which would be much more convenient, but it seems to have disappeared. For the path to the .HEX file, I should be able to use

or

but it appears the ‘final_output_path’ is a bit more directly related to the .HEX file location. So, to make this work using the VM post-build facility, the required ‘board.txt’ file should look like:

When the above is placed in ‘board.txt’ and the Arduino sketch is run, I got the following output:

Which appears to be correct, and the Teensy was obviously updated (got the USB disconnect/connect tones, and the LED tell-tale stopped and then resumed blinking). It looks like I might actually have a working program now – woo hoo!

06 October 2021 Update:

One minor snag occurred with the build property used to pass the COM port number to the command-line program; I had been using {serial.port.caption} and it turns out that this parameter (along with many other serial.port parameters) isn’t always available – things change depending on the COM port chosen in the Arduino program and the actual device connected (or not connected) to the PC. After another round of posts in the Visual Micro forum, Tim Leek pointed out that the ‘serial.port’ parameter should always be available. I had missed this on this because it isn’t grouped in with the rest of the ‘serial.xxx.xxx’ parameters. With the benefit of hindsight, I can see why this is so; the ‘serial.port’ parameter is probably pulled from the VS2019 IDE, while the other ‘serial.port.xxx’ parameters are hardware-specific. Once I switched my ‘board.txt’ to use the ‘serial.port’ parameter vs ‘serial.port.caption’, I started getting much more consistent results. Here’s the ‘final’ (I hope) board.txt configuration:

One other small snag; my ‘TeensySerialBridgeV1’ program was originally intended to just pass serial data back and forth between two terminal programs (Tera Term on one side, the Teensy sketch on the other). As a result, there was a built-in assumption that both serial connections would always be available, as shown in the following code snippet:

The first ‘while()’ statement causes the program to hang if the physical Teensy USB connection is missing. This gave me some grief when I removed the Teensy USB cable to prove I could do the entire update cycle with only the Serial1 hardware serial device connected – oops!

The final Teensy program is shown below:

09 October 2021 Update: The final product

After encountering and solving a few more ‘gotchas’, I think I now have a reliable Teensy OTA mechanism using the VS2019/Visual Micro IDE and Joe Pasquariello’s wonderful ‘FlasherX’ code . The ‘trick’ is using Visual Micro’s ‘board.txt’ feature to add a post-build command that runs each time the Teensy sketch is compiled, and a small C# command-line program to trigger the FlasherX update code and feed it the correct Hex file.

I created a very simple Teensy project called ‘TeensyOTADemo’ that does only two things; it flashes the built-in LED, and launches the FlasherX update code when triggered by a ‘U’ (for Update) ascii code on Serial1 (pins 0 & 1 on a T3.5). Then I created a small C# command-line program to actually send the trigger character and then transmit the .HEX file created as part of the normal compile process.

Then I created a ‘board.txt’ file containing a post-build command that launches the C# command-line program and feeds it two arguments – one containing the build path, and the other containing the COM port number to be used to connect to the Teensy’s Serial1 port.

Here’s the Teensy sketch:

And the C# command-line program:

And last, the ‘board.txt’ file contents that launches the C# command-line program:


Here’s a short video showing the update process. The clip starts with the ascii ‘U’ trigger code arriving at the Teensy Serial1 port, and ends with the update finished and the Teensy rebooted back into the ‘blink’ code.

09 October 2021 Update: HC-05 Bluetooth link:

The next step is to replace the FTDI module with an HC-05 Bluetooth link. I had some HC-05 modules from previous work, and I had done a similar project with an Arduino some time back – so how hard could it be?

I hooked up a HC-05 to the Teensy as shown in the following photo, using a bench power supply to power both the HC-05 and the Teensy 3.5

HC-05 and Teensy both powered by lap power supply

Then I paired the HC-05 with my laptop; this took a bit of fumbling around, as the laptop thought it was still paired with an HC-05 even though it’s been over two years since I last used it, and I had since forgotten that the default password was ‘1234’. In any case, I eventually got it paired, as shown, with an automatically generated COM port on COM4.

Next I fired up Tera Term and created a connection to COM4, as shown:

And then tried sending some ascii text through to the Teensy. This appeared to work fine, but when I tried a program update via the HC-05 link I got an error that indicated the FlasherX code didn’t like something I was sending it.

The ‘abort – bad hex line’ output happens before I try to send it the .HEX file contents, so it clearly isn’t choking on that; maybe some extra characters getting sent instead of just the “U” character?

Looking at the ‘FlasherX’ code, it appears that the offending characters/line is supposed to be printed out with the lines

which implies whatever caused the problem is non-printable – maybe an extra CR/LF? So, I added some debug code to update_firmware() to print out each line as it is read into the ‘line’ character array, so maybe I can see the problem.

Hmm, more mysteries. I reconnected the regular USB connection, and uploaded the new version of the sketch. Then I disconnected the USB connector, and used Tera Term to send the update command (“U”). However, this time the OTA Update code didn’t choke, and AFAICT performed a completely successful update via BT and the HC-05 – yay! Of course, I now have no idea what caused the problem to start with, and whether or not my addition of a few lines of debugging code made the difference.

So, I undid the changes to the update code and tried again. And, as shown below, the OTA update worked perfectly – double Yay!!

The printed out lines from the .hex file are from the previous run.

So now to try an OTA update using my command-line program instead of Tera Term. I disconnected TT from COM4, and then, after selecting COM4 in the VS2019 IDE, pressed F7 to start the process. This worked as well, but it appeared the transfer rate was considerably slower – no idea why.

Here’s a short video showing the OTA update using the command-line program and BT OTA.

Teensy OTA update using command-line program and HC-05 BT link

Just comparing the overall times from the two videos above the one using the command-line program runs approximately twice as long. It’s hard to believe it is only the use of the command-line program vs Tera Term that is causing the delay – but it is possible that TT was simply running at a higher baud rate – more study required.

In summary, however, slow rate or no, it is clear that we now can implement completely automated, seamless OTA update processes using VS2019 and Visual Micro. Now that the groundwork has been laid, I would assume that this same strategy could be implemented for other IDE’s as well.

I have started GitHub repositories for the Teensy demo sketch TeensyOTADemo and my TeensyFlash C# command-line program on my GitHub site, and I would welcome suggestions/improvements.

10 October 2021 Update:

I’m trying to determine what caused the slow update when using my command-line program and the HC-05 when doing an OTA update relative to the situation when using Tera Term to manually transfer the .HEX file to the Teensy. In both cases, the OTA link is through the HC-05, so that module shouldn’t be the cause. However, just to check I went back and checked the HC-05 settings using the ‘AT Command Mode’ trick discussed in this post. AFAICT, the HC-05 was set to 115200 baud rate. I also checked to see if I had somehow misconfigured the serial port created in my command-line program, but it is set to create a port at 115,200 BPS as well. So, I’m officially baffled.

To investigate the problem, did an OTA update via the HC-05 BT link, but using Tera Term to trigger the update process and send the .HEX file. Using this method, it took approximately 14 sec to transfer the entire 3142-line .HEX file. Then I did an OTA update using my command-line program, using the same HC-05 BT link. With this setup, the transfer took approximately 35 seconds – over twice the time. Watching with my scope, it is clear that the transfer takes place at a much slower rate when using the command-line program, even though (at least AFAICT) the programmed baud rates are identical in both cases. The two photos below show the transfer rate for both cases, using the same time scale for both.

File Transfer using Command-line Program and HC-05 BT Link
File Transfer using Tera Term and HC-05 BT Link

After a bit of web research on C# .NET serial port programming, I came across this post, which indicates that using .NET for this task was not a very wise move – bummer!

Stay tuned,

Frank