Tag Archives: Teensy

I2C between an Arduino Mega and a Teensy 3.x

posted 18 March 2020

This post describes my efforts to troubleshoot an I2C communications problem between an Arduino Mega control board and a Teensy 3.2 IR beam demodulation module.

Background:

My Wall-E2 autonomous wall-following robot homes in on a modulated IR beacon to connect to a charging station. The IR beacon modulation is decoded by a dedicated Teensy 3.2 and provides left/right and combined steering values to an Arduino Mega main controller over an I2C link.

During my recent work to update the robot after some enhancements, I discovered that the main controller was no longer receiving steering information from the Teensy, even though both seemed to be operating properly.  Initial efforts to troubleshoot the problem did not bear fruit, so I was forced to back up and start over from scratch.

Troubleshooting:

Initially I thought the problem was a loose connection, as the system was working before.  However, I am now pretty sure that I have eliminated all the obvious culprits, so I am left with the non-obvious ones.

To start with, I resurrected an old example project to demonstrate I2C master/slave operation between two Teensy modules.

Teensy 3.2 Slave (left) and Teensy 3.5 Master (right)

Here’s the Teensy ‘Master’ code:

And the Teensy ‘Slave’ code:

With this configuration I got the following output:

Master:

Slave:

So this seems to be working OK.

Then I added in a third Teensy module (T3.5) running my newly developed I2C-Sniffer code, so that I could ‘sniff’ the I2C traffic between the two Teensy’s.  This will give me a ‘known-good’ baseline for when I move back to the non-working Arduino Mega Master and  Teensy Slave condition that is causing me problems.

Teensy 3.2 Slave (left), Teensy 3.5 I2C Sniffer (middle), Teensy 3.5 Master (right)

Here’s the Teensy ‘Sniffer’ code:

With this setup with the master sending data to the slave, I got the following outputs:

Master

Slave:

I2C Sniffer:

In the other direction, with the slave sending data to the master, I got the following:

Slave:

Master:

I2C Sniffer:

Where the HEX sequence 4B D8 A9 AD converted to a IEEE float = 2.83984e+07

So, the Teensy-to-Teensy I2C connection is clearly working in both directions, and the Teensy I2C Sniffer is successfully capturing and decoding the I2C traffic between the two modules – cool!

So, the next step is to replace the ‘Master’ Teensy with an Arduino Mega and repeat the process.  Hopefully this will allow me to figure out why the slave-to-master data transfer isn’t working

21 March 2020 Update:

Based on the above results, I formed a hypothesis that the problem with sending I2C data from a Teensy slave to an Arduino Mega master might be due to the Mega not being able to properly interpret 0-3.3V transitions from the Teensy.  So, I decided to construct a low-to-high level converter to make sure that SDA data from the Teensy was presented to the Mega as 0-5V transitions rather than the Teensy’s ‘raw’ 0-3.3V ones.  To do this I used the circuit shown below, except with a 1K instead of 10K resistor for R2 to clean up the transitions at 100KHz.

Bidirectional 3.3-5V level shifter using 2N7000 MosFet

Here’s a scope photo of the 3.3V input to and the 5V output from the 2N7000-based level shifter. The top trace is the Teensy 0-3.3V output, and the bottom trace is the 0-5V level-shifted version.  Both traces are 1V/cm, and the horizontal time scale is 20 uSec/cm.

Teensy slave, Mega master: Top: 0-3.3V input. Bottom: 0-5V output. Both traces are 1V/cm vertical, 20 uSec cm horizontally

To verify that the above circuit worked properly, I used it in the SDA line (the SCL line doesn’t require any level shifting as it is from the 5V Mega to the 3.3V Teensy) with a Teensy slave and a Teensy master.  This is OK, as Teensy 3.x’s can handle 5V inputs, and it worked properly in both directions.  Here’s a shot of the SDA line

Teensy slave, Teensy master: Top: 0-3.3V input. Bottom: 0-5V output. Both traces are 1V/cm vertical, 20 uSec cm horizontally

However, after replacing the Teensy master with the Mega one, I had the same problem as before – I can successfully transfer data from Mega master to Teensy slave, but not in the other direction. There is still a problem somewhere, and I no longer think it’s due to level-shifting problems.  From the above scope photos, it is clear that only a few bytes are transmitted by the Teensy slave each time it services the OnRequest() interrupt, while that same function transmits MUCH more data when servicing the same request from the Teensy 3.5 master.

Stay tuned!

23 March 2020 Update:

OK, back to the basics:  I downloaded the code for a very basic Arduino-Arduino I2C tutorial, and verified that it worked OK.  I also took a scope shot of SDA/SCL activity during the two-way data transfers, as shown below:

Basic Arduino-Arduino tutorial layout

I2C line activity for the ‘Basic Arduino – Arduino I2C Tutorial’

So now that I have a working baseline for the Arduino-Arduino I2C case, I plan to incrementally modify it toward duplicating the non-working Arduino-Arduino I2C case until it breaks, and then I’ll know that the last incremental modification is the culprit. At the moment, I’m leaning toward the use of the I2CDev & SBWIRE libraries as they are the only significant difference between the working and non-working setups.  We’ll see….

24 March 2020 Update:

I created a new Arduino project called ‘I2C_Master_Tut_Mod1’ initially identical to ‘I2C_Master_Tutorial’ and verified that it worked properly with the unmodified ‘I2C_Slave_Tutorial’ project, and then started modifying it.

  • Added #include <PrintEx.h> //allows printf-style printout syntax
    StreamEx mySerial = Serial; //added 03/18/18 for printf-style printing, and changed all occurrences of ‘Serial’ to ‘mySerial’ and modified print statements to ‘printf’ format. All worked fine.
  • Added #include <I2C_Anything.h> and changed loop() to write a float value to the slave.  However, this caused a compile error.  When I started tracking it down I realized that I had previously modified ‘I2C_Anything.h’ to #include SBWIRE.h rather than the original #include Wire.h.  So, this could be the culprit, assuming that there is something about SBWIRE that causes Arduino I2C slaves to misbehave.
  • Copied I2C_Anything.h/cpp from the Libraries folder to my new ‘I2C_Master_Tut_Mod1’ folder, and un-modified it to reference back Wire.h vs SBWIRE.h.

25 March 2020 Update:

As it turns out, I was never able to get the ‘I2C_Master_Tut_Mod1’ project working with #include <I2C_Anything.h> , no matter what I did, including copying the #include <I2C_Anything.h> to the local folder and including it as a resource int the VS2019 project, build cleans, removing the _vm folder created by Visual Micro, etc.  To make it even stranger, I created a new Arduino project in VS2019 called ‘I2C_Master_Tut_Mod2’, copied the ‘I2C_Anything.h’ file to the local folder, and it compiles without error!  So now I have two almost completely identical Arduino projects, one of which compiles fine and the other of which blows a bunch of errors about twi.c.  I’m currently working with Tim Leek at VisualMicro to sort out what I did wrong.

27 March 2020 Update:

Ok, I now have the Arduino Mega master to Arduino Uno slave setup working, with I2C_Anything.  I can transfer a float value in either direction with no problem. The master and slave code sets and the corresponding outputs are shown below.

Master Sketch:

Slave Sketch:

Master  & Slave Output:

So now I have a working Arduino-Arduino I2C master/slave setup, using I2C_Anything.h to transfer float values back and forth, and a working Teensy-Teensy I2C master/slave setup, using I2C_Anything.h to transfer float values back and forth. 

The next step was to replace the Arduino Uno I2C slave with the Teensy 3.2 I2C slave and see if I can get that combination working.  This turned out to be surprisingly easy – basically plug-and-pray (oops, I meant ‘play’).

Now that I have a simple Arduiono Mega Master – Teensy 3.2 Slave setup working, I can start to explore why my 4WD robot setup doesn’t work.  Some possibilities:

  • It could be that the SBWire version of the Wire library has some hidden bugs in the slave-related code.  The SBWire library eliminates the well-known and well-hated ‘hang’ bug in the Arduino wire library.
  • It could be that some other library I am using, like the RTC library or the Adafruit FRAM library is interfering with the Arduino-Teensy interrupts needed for master-slave communication.
  • It could be that the IR demod program running on the slave Teensy is somehow interfering with master-slave communications (this is very unlikely as this same program had been working flawlessly for several months).
  • Something else…

To start with, I copied the current ‘I2C_Master_Tut2’ VS2019 project to a new one – ‘I2C_Master_Tut3’ before making any modifications.  I plan to keep a ‘breadcrumb trail’ of incrementally modified projects so that I can go backwards in case I get lost at some point (and, in my fairly long 50+ years as a practicing engineer, I have learned that ‘getting lost’ is part and parcel of any non-trivial troubleshooting project).

Once I verified that ‘Tut3’ properly communicated with the Teensy slave, I started making modifications.

  • Replaced #include <Wire.h> with #include SBWire.h.  This failed in compile with a linker error, but succeeded after I did a ‘Build -> Clean’.  After uploading the new ‘Tut3’ version to the Mega, I found that master-slave communications still worked properly.  This is actually a welcome development, as that now eliminates SBWire as the culprit for lost Mega-Teensy I2C capability on the robot.
  • Replaced ‘#include “SBWire” with #include “MPU6050_6Axis_MotionApps_V6_12.h” and include “I2Cdev.h” (this includes SBWire.h).  These two libraries are required for interfacing to the MPU6050 IMU module.

29 March 2020 Update:

So I created a new copy of the I2C master, ‘I2C_Master_Tut4’ and started adding things from the original non-working robot program.

  • Added all the remaining #include’s.  Still works fine with the Teensy 3.2 slave
  • Added all the #defines. Still works fine
  • Added all the IRHOMING parameters.  OK
  • Added all ENUMs, Battery constants, distance measurement support constants & array declarations,  motor parameters, wheel direction constants and variables, and heading based turn parameters.  No problem
  • Added pin assignments.  No problem.
  • Added all the other pre-setup stuff.  No problem
  • Added ‘ Serial1.begin(9600); //03/04/16 bugfix’. This isn’t actually used anymore in the robot project, but it is there, so it could be the culprit.  Nope –  master/slave comms still OK.
  • Added RTC initialization and  support function.  Still no problem
  • Added FRAM initialization.  No problem.
  • Added MPU6050 initialization.  No problem
  • Added PID distance array and incremental variance initialization.  No problem
  • Added I/O pin initialization.  No problem
  • Added the rest of Setup(), and all the support functions required for the POST check to run.  Compiles and runs fine.  Of course, no peripherals are attached, so not much happens.

At this point I have all the pre-setup and setup code incorporated into the master/slave example, and everything is still happily plugging away.  So obviously the culprit hasn’t yet been identified.  However, before going any farther, I think I’ll drop ‘Tut4’ into the safe-deposit box and create a ‘Tut5’ to continue on into the loop() function.

OK, so I have a ‘Tut5’ project now, with everything up to loop() incorporated from the ‘FourWD_WallE2_V2’ robot project.  Now the question is, ‘What next?’.  I need to figure out what is causing I2C comms between the Mega master and the Teensy 3.2 slave to fail, so I need a way to faithfully replicate/simulate/emulate the actual robot code for this function.

The current robot algorithm as pertaining to the Teensy IR Demod module

  • Each time through the loop, the current operating mode is determined.
    • If IsIRBeamAvail() returns TRUE, the IR HOMING mode is activated
      • IsIRBeamAvail() gets three float values from the Teensy 3.2, and returns TRUE if the total of the first two values (Fin1 & Fin2) is above a set threshold.  It is this function that is failing to communicate with the Teensy.  More specifically, it is this function that is not successfully acquiring the three float values, and subsequently always returns FALSE.

So, it may be that all I have to do is to call IsIRBeamAvail() by itself, and modify the slave code to send back three floats as expected.  If this works, then I’ll have to start suspecting the robot’s Teensy or the I2C wiring, or something else entirely.

2 April 2020 Update:

After some additional fumbling around with I2C_Anything and the PrintEx library printf() formats, I now have a working ‘IsIRBeamAvail()’ function in ‘Tut5’, as follows:

Which, when connected to my basic  Teensy 3.2 I2C slave program produces the following (correct) output:

At this point we have a working robot code emulation that communicates successfully with a basic Teensy 3.2 slave.  So, the available culprits have been reduced significantly to

  • Something about the Teensy IR demod code
  • The I2C Wiring
  • A hardware problem at either the robot’s Mega controller or the robot’s Teensy 3.2 controller
  • Something else.

For the next step, I modified the Teensy IR Demod code to emulate the IR detector response and loaded this code into the Teensy 3.2 slave connected to the Mega master.  After some initial mis-steps, it started working nicely, with the following (correct) output from the Mega master:

So this eliminates the ‘Something about the Teensy IR demod code’ possibility from the above list.

Next, I replaced the direct SCL/SDA jumpers with the daisy-chain wiring from the robot, and miracle of miracles, I2C comms failed – YAY!!.  Now hopefully I can figure out (and fix) the problem.

  • First, I un-replaced the daisy-chain wiring to confirm that I2C comms were still working, and they were.
  • Unplugged the daisy-chain from the FRAM, 6050IMU, and RTC modules – now working OK
  • Plugged back in to FRAM unit (now have FRAM and Teensy 3.2 in chain)- – still OK
  • Plugged back in to 6050 IMU (now have IMU, FRAM, and Teensy 3.2 in chain)- failed
  • Plugged back in to RTC (now have RTC, FRAM, and Teensy 3.2 in chain)- failed

3 April 2020 Update:

The next step in the saga was to load the ‘Tut5’ code onto the robot’s Mega 2560 controller, with the slave code still running on the original (off-robot) Teensy 3.2.  Setting things up in this fashion allowed the sensors to receive power from the robot’s Mega controller as in normal operation.

In the photo above, the Teensy 3.2 slave is shown to the left of the robot.  The I2C ‘daisy-chain’ cable starts at the Mega controller (just to the right of the front left wheel), and goes through three ‘hops’ to the Teensy. With this setup, the I2C comms code still ran fine, with direct I2C jumper wires or with the daisy-chain wiring setup (with no connections to the other I2C sensors) as shown above.

The next step was to connect the robot’s Mega controller to the robot’s Teensy 3.2 running the unmodified IR Demodulation code, with the I2C daisy-chain cable, but without anything else connected.  This actually worked!  So now I have a working robot system again, so something about the connections with the other sensors must be killing the I2C link to the Teensy.  Here’s a scope photo of the SDA line showing the data activity. The vertical scale is 1V/cm, showing the HIGH value is about 3.8V, so the Mega must be comfortable with that value as a HIGH logic level

In the above photo, notice the business card for Probe Master, Inc (www.probemaster.com).  These folks were kind enough to replace my ten year old scope probe with a new upgraded version at no cost – thank you Probe Master!

At this point I have the ‘Tut5’ program running on the Mega, and the unmodified IR Demod code running on the robot’s Teensy 3.2, and the Mega appears to be acquiring valid demod data from the Teensy.  To further validate the data, I fired up my charging station with it’s square-wave modulated IR beam, and was pleased to see that the acquired data from the Teensy varied appropriately as I manually rotated the robot, as shown in the following Excel chart.

So, after all this work, the whole thing came down to a bad I2C ‘daisy-chain’ cable. This particular cable has been on the robot for a while, and was the soldering graduation exercise of my grandson Danny  when he was here a couple of summers ago.  It wasn’t the best job I’ve ever seen, but it was pretty good for a 15 year old ;-).  In any case, I took the opportunity to build a new cable with smaller diameter wire so things would fit a little bit better into the Pololu pins, sockets, and 2-pin header sleeves as shown in the following photo.

New I2C ‘daisy-chain’ cable.  Run starts from the Mega connector at bottom left, then to the RTC, IMU and FRAM modules in that order, then last to the Teensy 3.2 IR Demod module.

05 April 2020 Epilogue:

Well, there was one last ‘gotcha’ in all this.  When I loaded the original ‘FourWD-WallE2_V2.ino’ program back onto the Mega, it still refused to acquire valid data from the Teensy IR Demod module.  So, I compared the ‘Tut5’ code to the ‘_V2’ code, and noticed three significant differences:

  • In the ‘_V2’ code, the ‘Fin1/2’ variables had at some time been changed from ‘long’ to ‘int’. While this sounds reasonable, it isn’t – because a Teensy ‘int’ is 4 bytes – the same as a Mega ‘long’.
  • In the ‘_V2’ code, the  Wire.requestFrom() call asks for 32 bytes, but the Teensy only sends 12.
  • In the ‘_V2’ code there is a loop that waits for either a timeout or receiving an entire 32-byte buffer from the Teensy.  Apparently the Teensy (and/or the Mega) is slow enough so that Wire.Available() never reports a non-zero buffer size, so the loop times out.

Replacing the line

with the line

did the trick, but I have no idea why.

Replacing the debug printout lines

with the line

Produced lines like this one from the ‘_V2’ program

Which appear to be the correct values for the given robot orientation with respect to the charging station.

With no other changed except removing power from the charging station, the output became

showing definitively that the ‘_V2’ program correctly identified and decoded the square-wave modulated IR beam.

So, after a two-week off-road trip to I2C comms hell and back, I think I finally have that particular issue put to bed.  It wasn’t exactly what I wanted to do, but it was at least interesting, and more important – consumed a lot of coronavirus quarantine time ;-).

Stay tuned!

Frank

Arduino SPI Data Exchange Between Two Arduinos in a Master/Slave Configuration

posted 13 March 2020,

While updating my four wheel autonomous wall-following robot (aka Wall-E2), I ran into a roadblock when I couldn’t get my Teensy 3.2 IR demodulator/tracker module to communicate with the Mega 2560 main microcontroller over I2C.  This worked fine when I last tested it, but now it seems to have taken a vacation.  It’s been a while, and I have added some functionality since I originally installed and tested IR homing to the charging station, but it still should all work, right?

In any case, after trying (and failing) to get the I2C bus connection working, I thought I might try an alternate solution and just use SPI between the main controller (Mega 2560) and the IR homing controller (Teensy 3.2).  This would have the advantage of making the Teensy independent of the I2C bus, and also give me the chance to play with a part of the Arduino ecosystem that I haven’t used before.

As usual, I started this process with a lot of Googling, and quickly ran across Australian Nick Gammon’s  “SPI – Serial Peripheral Interface – for Arduino  .  This post was way more than I ever wanted to know about SPI, but it sure is complete!

Then I moved on to trying to get SPI working for myself.  As noted above, my intended application is to transfer steering values from my IR detector/demodulator/homing module to the main Mega 2560 microcontroller. The Mega uses the steering data to adjust wheel speeds to home in on a charging station, so Wall-E2 can continue to roam our house autonomously.

It turns out that SPI between two Arduinos isn’t entirely straightforward, at least not at first.  I went through a bunch of iterations, and even more passes through the available documentation.  In the end though, I think I wrassled it into a reasonable facsimile of a working solution, as shown below.  The program repeatedly transfers a fixed string (“Hello World!”) from the UNO (master) to the Mega 2560 (slave), and then transfers the string version of a float value (3.159) from the slave to the master.

The Circuit:

The circuit and layout is just about as basic as it gets.  The UNO (master) connections are the default pinouts:

UNO (master) connected to Mega 2560 (slave) using default SPI pins on both ends

 

The Master:

The Slave:

The above configuration produced the following output:

Master:

 

Slave:

Now that I’ve had my fun and games with SPI, I’m still not at all sure I want to use it to connect the Mega to the Teensy IR Demodulator/Homing module.  While it is almost certain to work (eventually), it has some drawbacks

  • It requires 4 additional wires
  • It requires that I add an ISR and other code to the Teensy module, and companion code to the Mega
  • More things to go wrong.

So, I think I’ll try again to get the original I2C based code working, as it worked before, so it should work again.  If I can’t get it working after another good try, I’ll go with door #2 (SPI)

Stay tuned!

Frank

 

Back to the future with Wall-E2. Wall-following Part VIII

Posted 25 January 2020

About 6 weeks ago I posted that I had finally killed the “intermittent MPU6050 failure” dragon, by belatedly following Pololu’s recommendations for installing bypass capacitors on their metal-geared motors.  Unfortunately it turned out that my celebration was cut short by more annoying intermittent MPU6050 failures, so I was once again forced back to the drawing board.

This time I decided that the only way to figure out what was going on was to actively examine the I2C traffic in real time, to determine who exactly was doing what to whom.  So, over the course of the six week period between my last declaration of victory to this one, I created a Teensy based I2C bus ‘sniffer’ and used it to figure out what was going on.  I was able to determine that the ‘master’ micro-controller continued to operate normally through a failure, but the MPU6050 didn’t. I was also able to determine that just resetting the IMU would not allow the system to recover, but resetting the micro-controller often did.   Moreover, I was able to definitively show that the problem was caused by ‘contact bounce’ on one or more of the four 6″ male-male jumper wires connecting the micro-controller to the IMU.  Eliminating these jumpers also (I hope) eliminated the last piece of the “I2C Intermittent failure” puzzle.

Looking back over the entire I2C failure saga, I now realize that this was the classic case of multiple failure modes complicating the troubleshooting effort.  The RFI/EMI problem caused by the Pololu metal-geared motors completely overshadowed the issue of non-secure jumper connections. Then, after finally coming to my senses and installing the recommended bypass capacitors on the motors, the ‘contact bounce’ problem was unmasked.  I do love interesting problems, but this one went past ‘interesting’ and was well into ‘agonizing’ by the time I got it solved ;-).

After getting everything set up, I ran some wall tracking tests in my entry hall ‘test range’ with pretty good results, as shown in the short video clip below.

Stay tuned,

Frank

30 January 2020 Update:

Still having trouble with the initial approach to a wall from outside the target distance. The robot still has a tendency to dive into the wall, unable to cope with the problem of the measured distance increasing instead of decreasing when the robot turns into the wall.  This inverse relationship makes it almost impossible to use a simple ‘turn toward the wall and wait for the distance to count down’ technique.

After thinking about this for while, I realized that this would all be so much simpler if I cheated and started with the robot placed parallel to the target wall.  Then the robot could simply turn 45 deg toward the wall and proceed until the measured wall distance was appropriate (Dtgt / 0.707), and then turn parallel again.  Then I realized that I could easily determine the parallel condition by turning the robot toward and/or away from the wall while continuously measuring the distance; when the measurement goes through a minimum, then the robot is parallel to the wall.  Simple in concept, and not all that hard to program, either.

 

IMU Motor Noise Troubleshooting, Part III

Posted 19 January 2020

In Part II of this saga, I described my continuing efforts to track down and fix the problem of intermittent failures associated with the MPU6050 IMU on my robot.  The MPU6050 IMU is required for the ability to make precise heading-based turns, which is in turn required to track walls at a designated stand-off distance.

This post summarizes the work to date and suggests new avenues of investigation for fully addressing the motor noise issue.

Summary of work to date:

  •  July 2019: First started working with heading-based turns, and first noticed the motor noise problem.  Basically the problem presented itself as frequent, abrupt, and wildly divergent heading readings when the motors were running, but perfectly stable readings when the motors are not running.  See this post for the details.
  • October 2019: Successfully demonstrated polling-based (vs interrupt-based) MPU6050 IMU management. This development meant that I could acquire yaw (heading) values on an as-needed basis rather than at a 20 or 200Hz rate, throwing away 99% of the results.  This was demonstrated in this post.
  • November 2019: Made another run at solving the motor noise problem using a home-brew optical isolator and  a 2-stage power filter.  After a LOT of work, I wound up discovering that most (but not all!) of the problem could be addressed with proper RF bypassing at the terminals of the metal-geared Pololu motors I was using.  See this post for the details.
  • Early December 2019:  Demonstrated heading-based wall offset tracking using my 2-motor robot, with RF bypassing installed on both Pololu metal gear motors.  No IMU failures were noticed during these runs.  See this post for details.
  • Early December 2019:  Reprised some of the motor driver testing performed back in May of 2019 (see this post), and again noticed MPU6050 IMU communication failures when the motors were running, but none when the motors weren’t running. This test was performed on the 2-motor robot using the Pololu motors with the RF bypassing in place. So clearly just the bypassing was not of and by itself sufficient to solve the problem; something else had to be going on.  See this post for the details.
  • Late December 2019 to mid-January 2020:  I decided I needed a tool to monitor the I2C bus traffic between the robot’s controller and the MPU6050 IMU – an I2C ‘sniffer’.  After some research, I found that the cheapest commercial sniffer cost about $330, and DIY sniffers were few and far between. I did, however, find a Teensy-based sniffer program by Kito, so I had a starting place.  After three major development stages, I had a Teensy 3.2 program that would reliably monitor I2C communications between an Arduino (Mega or Uno) master and a MPU6050 slave, using the polling approach developed earlier.  See this post, this post, and this post for the development details.

Current Effort:

With the above history in mind, I applied my new I2C sniffer tool to the Motor Noise Problem.  As usual, I started this using the simplest possible setup; an Arduino Mega acting as the I2C master running my polling based ‘MPU6050_MotorNoiseTest1’ program, and a Teensy 3.2 and a MPU6050 IMU module both mounted on a small plugboard, as shown below.

Arduino Mega I2C master, with Teensy I2C sniffer and MPU6050 module on a separate plugboard

I played around with this setup for a while, and captured at least one IMU communications failure with the sniffer active. The failure occurred when I was moving the plugboard around a bit to verify that the MPU6050 IMU heading values changed appropriately.  At some point I noticed the I2C monitor output had changed its character significantly, so I quickly stopped the sniffer program and opened the log file (see attached file below).

From the log I can see that things proceeded normally until 6012443 mSec ( 1.67 hours) and then changed to report that nothing was being received from register 0x72 (the FIFO count register). This continued until 6022224 mSec (9.8 seconds later) where it returned to what looks like normal operation.

So, my preliminary guess at what happened is the connection from the Mega to the Teensy/MPU6050 got dropped momentarily, and it took the Teensy a while to find another START sequence in the I2C data stream from the Mega, as the ‘2048’ number in “6017280: processed = 2048 elements in 3 mSec” means that the capture buffer overflowed before a START sequence was detected.  “At 6022240: processed = 1224 elements in 2 mSec” means that a normal Mega ‘burst’ was captured and operation returned to normal.

Since the Teensy I2C monitor is on the MPU6050 end of the male-male jumpers, It begins to look like the Mega was still doing fine, but the jumper connection burped on one end or the other.  More testing to follow.

Loader Loading...
EAD Logo Taking too long?

Reload Reload document
| Open Open in new tab

Download

 

Next, I moved the plugboard containing the Teensy I2C Sniffer and the MPU6050 module to my 2-motor robot, and used the existing Arduino Uno on the robot as the master, as shown below.

I loaded my MotorNoiseTest1 program on the Uno, and allowed it to run both motors at a steady rate, while monitoring the I2C traffic with the Teensy, and also monitoring the heading values being printed out by the Uno.  I started the program just before 1PM, and it was still running fine with no IMU errors at 10pm, more than 9 hours later!  The I2C sniffer log shows regular communication with the MPU6050, and the calculated yaw value based on the packet bytes received by the sniffer program matches the yaw value calculated in the Uno program. This is clear verification that the sniffer program will run ‘forever’, and that at least in this case, the two motor robot will also run ‘forever’ with no  yaw errors.

Based on my earlier experience with the captured I2C communications failure, I’m more inclined now to believe that motor vibration or other mechanical perturbation is causing a momentary I2C bus or power/ground lead disconnect.  More tests to follow:

21 January 2020 Update:

After the 10-hour run described above, I tried to induce some failures by fiddling with the I2C and power/ground jumper wires, and found that I could easily and reliably cause a failure by ‘flicking’ the wires with my finger or a pen.  After each failure, the built-in recovery routine of clearing the FIFO and resetting the DMP failed to restore communications.  However, manually resetting the UNO did allow the system to recover.

From the above, I believe it’s safe to say that the current male-male jumper connections between the UNO and the Teensy/IMU are unreliable, and are hopefully the only remaining failure mode.  I haven’t quite figured out how to replace the connections with something more reliable, but I’m working on it.  I moved the IMU module from the plugboard and plugged its I2C pins directly into the I2C sockets on the UNO.  Then I replaced the power & ground leads with a permanent twisted pair connection to the Wixel shield, as shown in the following photo.

MPU6050 plugged directly into Uno board, with pwr/gnd jumpers replaced with permanent twisted pair

Then I fired up the system and ran it for a while but was unable  to make it fail.  This is encouraging news to say the least.

Stay tuned,

Frank

Teensy I2C Sniffer for MPU6050 Part II

Posted 13 January 2020,

In my last post on this subject, I described my efforts to build an I2C bus sniffer using a Teensy 3.2 micro-controller.  This post describes my efforts to move from a fixed array containing a 928-byte snapshot of an I2C bus conversation between an Arduino Mega 2560 and a MPU6050 IMU to a live, repeated-burst setup.

As the source for I2C traffic for the MPU6050 IMU I am using my MPU6050_MotorNoiseTest1 Arduino project with no motors or sensors connected.  All the code does is ask the MPU6050 for a yaw value every 200 mSec (the value of NAV_UPDATE_INTERVAL_MSEC), as shown below:

The Teensy code to monitor the I2C bus traffic is shown below.  When I first started working with this project, I copied Kito’s I2C sniffer code, which used Teensy’s Timer1 interval timer set to produce interrupts every 1 uSec, and an ISR to capture the data.  This turned out to be hard to deal with, as I couldn’t add instrumentation code to the ISR without overrunning the 1 uSec interrupt period, leading to confusing results.  So, for this part of the project I disabled the Timer1 interrupt, and called the ISR directly from the loop() function.  As others have pointed out, the Arduino loop() function does a lot of housekeeping in the background, so for top performance it is best to never let loop() execute, by placing another infinite loop inside loop() or inside setup().  This is what I did with the code designed to investigate whether or not the Teensy could keep up with an I2C bus running at 100Kbs.

The ‘capture_data()’ function (no longer used as an ISR) captures SCL & SDA states with a single port operation as shown

and then everything from a START pair (0xC followed by 0x4) to a STOP pair (0X4 followed by 0xC) inclusive is captured in the raw_data array.

Any I2C Sniffer project like this one assumes that I2C activity occurs in short bursts with fairly long pauses in between.  This is certainly the case with my robot project, as yaw data is only acquired every 200 mSec.  However, there is still the problem of determining when a I2C ‘burst’ has finished so the sniffer program can decode and print the results from the last burst.  In my investigation, it became clear that at the end of the burst both the SDA line goes HIGH and stays that way until the next START condition (a 0XC followed by a 0X4).  So then the question becomes “how many 0XC/0XC pairs do I have to wait before determining that the last burst is over?”

In order to answer this question I decided to use my trusty Tektronix 2236 O’Scope and Teeny’s ‘digitalReadFast’ and ‘digitalWriteFast’ functions to implement a hardware-based timing capability using Teensy pins 0,1, and 2 (MONITOR_OUT1, 2 & 3 respectively).  Among other things, this allowed me to definitively determine that a ‘idle’ (0XC/0XC) count of 1000 was too small, but an idle count of 2500 was plenty, without consuming too much of the available processing time.  It also turned out that ‘idle’ counts all the way up to 30,000 work too, but leave less time for processing.

O’Scope shot showing I2C traffic on the bottom trace, and the point at which 2500 0xC/0xC (Idle) pairs is reached on the top trace (the high-to-low transition)

As can be seen in the above photo, the I2C ‘sentence’ lasts about 15 mSec, and the ‘idle’ condition is detected about 5 mSec later for a total of about 20 mSec out of the nominal 200 mSec cycle time for my robot application. This leaves about 190 mSec for I2C sentence processing and display.

18 January 2020 Update:

Success!!  I now have a working Teensy 3.2 I2C Sniffer program that can continuously monitor the I2C traffic between my Arduino Mega test program acting as a I2C master and a MPU6050 IMU I2C slave.   The Teensy code is available on my GitHub account here.

A major challenge in creating the sniffer program was the requirement to sample the I2C SCL & SDA lines quickly enough to accurately detect the line transitions denoting all the different I2C signals.  With the I2C bus running at 100Kbs, SCL (clock) transitions occur every 5 uSec. Good sampling requires at least 2 and preferably more samples per SCL state.  As noted above, I started off by copying the ISR routine from Kito’s I2C sniffer, but discovered I needed to add some logic to zero in on the desired I2C bus states (IDLE, START, DATA & STOP), and the additional code made the ISR take more than the desired 1 uSec window.  After posting about this problem to Paul Stoffregen’s Teensy forum, I got some good pointers for speedup, incuding a post that mentioned the Teensy FASTRUN macro that runs functions from RAM rather than FLASH. As it turned out, adding this macro to the program allowed me to reduce the ISR cycle time from about 1.4 uSec to about .89 uSec – yay!  The final ISR routine is shown below:

Note the use of digitalWriteFast() calls to output timing pulses on Teensy hardware pins so I could use my trusty Tek 2236 100 MHz O’scope to verify proper timing.

Once I got the ISR running properly, then I focused on getting the data parsing algorithm integrated into the program.  I had previously shown that I could correctly parse simulated I2C traffic, so all the current challenge was to integrate the algorithm in a way that allowed continuous capture-decode-print cycles at at rate that could keep up with the desired 5 measurements/sec rate.  So, I instrumented the sniffer program to display the decoded IMU traffic, along with the calculated yaw value and the time required to perform the decode.

Here’s a short section of the printout from the test program, showing the time (in minutes), the yaw (relative heading ) value retrieved from the IMU, and left/right ping distances (unused in this application).

And here is the corresponding output from the I2C sniffer program

In the above printout, each printout shows the individual transmit & receive ‘sentences’ to/from the IMU, and the 28-byte packet received from the IMU containing, among other things, the values required to calculate a yaw (relative heading value).  As can be seen, the yaw value calculated from the received bytes, closely matches the yaw values retrieved using the test program.  In addition the last line of each section of the readout shows the time tag for the start of the decode process, and the total time taken to decode all the bytes in that particular burst.  From the data, it is clear that only 1-2 mSec is required to decode and display a full burst.

The complete I2C Sniffer program is available on my GitHub site here.  The complete test program that obtains a yaw value from the IMU every 200 mSec is shown below:

The above program was intended to help me troubleshoot the intermittent MPU6050 connection failures I have been experiencing for some time now.  The purpose of the new I2C sniffer project is to create a tool to log the actual I2C traffic between this  program and the IMU. The idea is that when a failure occurs, I can look back through the sniffer log to see what happened; did the Arduino Mega stop transmitting requests, or did the IMU simply stop responding, or something else entirely.

 

 

Teensy I2C Sniffer for MPU6050

Posted 02 January 2020

On my last post on the I2C subject, I described an Excel VBA program to parse through a 928-byte array containing the captured I2C conversation between an Arduino Mega and a MPU6050 IMU module. The Arduino was running a very simple test program that repeatedly asked the MPU6050 to report the number of bytes available in its FIFO.  Then I used Kito’s I2C sniffer code to capture the SDA/SCL transitions, which I then copy/pasted into Excel.

This post describes the next step, which was to port the Excel VBA code into a Teensy sketch using the Arduino version of C++, moving toward the ultimate goal of a Teensy based, fast I2C sniffer that can be used to monitor and log long-term (hours to days) I2C bus conversations to determine what is causing the intermittent hangups I’m seeing with my Arduino Mega/MPU-6050 robot project.

The code port took a while, but mostly because of my own lack of understanding about the details of the I2C protocol and the specifics of the communication between the Arduino test program and my MPU6050 IMU module.  After working through these problems, the end result was surprisingly compact – less than 500 lines, including 50 lines of test data, LOTS of comments and debugging printouts, as shown below:

When I ran this program against the captured data, I got the following output:

which, when compared against the debug printout from Jeff Rowberg’s I2CDev program,

shows that my Teensy program correctly decoded the entire test dataset.

The next step in the process is to modify the above program to allow long-term real-time monitoring and logging of a live I2C bus. By ‘long-term’, I mean hours if not days, as the object of the exercise is to figure out why the I2C bus connection to the MPU6050 on my two-motor robot intermittently fails when the motors are running.  A failure can occur within minutes, or only after several hours, and there doesn’t seem to be any rhyme nor reason, except that the motors have to be running.

In normal operation, my two-motor robot obtains a heading value from the MPU6050 once every 200 mSec or so.  This I2C bus activity might comprise only 100 SCL/SDA transitions or so, and no other I2C bus activity takes place in the times between heading value requests.  So, there will be few mSec of burst activity, followed by a 150-190 mSec idle period.   To monitor and log in real time, I need some sort of FIFO arrangement, where the I2C transition data can be saved into the FIFO during the burst, and then processed and saved into a log file during the idle period.

While I was searching the web for I2C sniffer code, I also ran across this thread by tonton81 describing his template based circular buffer library for the Teensy line.  The thread started a little over two years ago, but has been quite active during that period, and tonton81 has made several bugfixes, updates, and enhancements to his code.  This might be just the thing for my project.

06 January 2020 Update:

After integrating tonton81’s circular buffer library into the project (thanks tonton81 for being so responsive with bugfixes!), I was able to demonstrate that the circular buffer version, when run against the same 928-byte simulated dataset, produced the same output as before, as shown below:

From the output above, it is clear that the Teensy can parse and print a typical 1000-byte burst in just a few mSec (3 in the above run), so it should have no problem keeping up with a 200 mSec data burst interval, and should be able to keep up with burst intervals down to around 10 mSec (100 bursts/sec!)  I suspect that the Teensy’s major problem will be not dying of boredom waiting for the next burst to process!

Here’s the full code (not including circular_buffer.h):

The next step in the project will be to modify the code (hopefully for the last time) to capture and process live I2C traffic bursts in real time.

I modified the interval processing code in loop() to reset the stop/restart Timer1 & clear FIFO each time the interval block is executed, and then reduced the processing interval from 200 to 50 mSec, to produce the following output:

The modified code:

and a short segment of the output:

Note that the processing block is indeed called every 50 mSec, and takes only a few mSec to complete.  The following is an O’scope image showing multiple 50 mSec periods.  As can be seen on the image, there is still a LOT of dead time between 928-byte bursts.

Top trace toggles at 50 mSec intervals to simulate periodic IMU data retrieval. Bottom trace shows IMU/MCU I2C communication bursts.

Stay tuned!

Frank

 

 

I2C Bus Sniffing with Excel VBA

In my never-ending quest to figure out why my I2C connection to an MPU6050 dies intermittently, I decided to try and record the I2C bus conversation to see if I can determine if it is the MPU6050 or the microcontroller goes tits-up on me.

Of course, this adventure turned out to be a LOT more complicated than I thought – I mean, how hard could it be to find and run one of the many (or so I thought) I2C sniffer setups out there in the i-verse?  Well, after a fair bit of Googling and forum searches, I found that there just aren’t any good I2C sniffer programs out there, or at least nothing that I could find.

I did run across one promising program; it’s a Teensy 3.2 sniffer program written by ‘Kito’ and posted on the PJRC Teensy forum in this post.  I also found this program written for the Arduino Mega.  So, I created a small Arduino Mega test program connected to a MPU6050 using Jeff Rowberg’s I2CDev library.

This program sets up the connection to the MPU6050 and then once every 200 mSec tests the I2C connection, resets the FIFO, and then repeatedly checks the FIFO count to verify that the MPU6050 is actually doing something.

When I ran Kito’s I2C sniffer program on a Teensy 3.2 (taking care to switch the SCL & SDA lines as Kito’s code has it backwards), I get the following output

which isn’t very useful, when compared to the debug output from Jeff Rowberg’s I2CDev program, as follows:

As can be seen from Jeff’s output, there is a LOT of data being missed by Kito’s program. It gets the initial sequence right (S,Addr=0x68,W,N,P), but skips the 8-bit data sequence after the ‘W’, and mis-detects the following RESTART as a STOP.  The next sequence (S,Addr=0x68,R,N,P) is correct as far as the initial address is concerned, but again omits the 8-bit data value after the ‘R’ direction modifier.

Notwithstanding its problems, Kito’s program, along with this I2C bus specifications document  did teach me a LOT about the I2C protocol and how to parse it effectively. In addition, Kito’s program showed me how to use direct port bus reads to bypass the overhead associated with ‘digitalRead()’ calls – nice!

I got lost pretty quickly trying to understand Kito’s programming logic, so I decided I would do what any good researcher does when trying to understand a complex situation – CHEAT!!  I modified Kito’s program to simply capture the I2C bus transitions associated with my little test program into a 1024 byte buffer, then stop and print the contents of the buffer out to the serial port.  Then I copy/pasted this output into an Excel spreadsheet and wrote a VBA script to parse through the output, line-by-line. By doing it this way, I could easily examine the result of each script change, and step through the script one line at a time, watching the parsing machinery run.

Here’s a partial output from the data capture program:

So then I copy/pasted this into Excel and wrote the following VBA script to parse the data:

The above script assumes the data is in column A, starting at A1. A partial output from the program is shown below, showing the first few sequences

The above output corresponds to this line in the debug output from Jeff Rowberg’s I2Cdev code:

So, the VBA program is parsing OK-ish, but is missing big chunks, and there are some weird 1 and 2 bit sequences floating around too.

After some more research, I finally figured out that part of the problem is that the I2C protocol allows a slave device to pull the SCL line low unilaterally to temporarily suspend transmissions until the slave device catches up.  This causes ‘NOP’ sequences to appear more or less randomly in the data stream.  So, I again modified Kito’s program to first capture a 1024 byte data sample, and then parse through the sample, eliminating any NOP sequences. The result is a ‘clean’ data sample.  Here’s the modified Kito program

and a partial output from the run:

After processing all 1024 transition codes, 96 invalid transitions were removed, resulting in 928 valid I2C transitions.

When this data was copy/pasted into my Excel VBA program, it was able to correctly parse the entire sample correctly, as shown below:

This corresponds to the following lines from Jeff’s program:

Although the VBA code correctly parsed all the data and missed nothing, there is still a small ‘fly in the ointment’; there is still an extra ‘0’ bit after every transmission sequence.  Instead of

we  have

with an extra ‘0’ between the ACK/NAK and the RESTART.  This appears in every transmission sequence, so it must be a real part of the I2C protocol, but I haven’t yet found an explanation for it.

In any case, it is clear that the Excel VBA program is correctly parsing the captured sequence, so I should now be able to port it into C++ code for my Teensy I2C sniffer.

Stay tuned!

Frank

 

 

 

 

 

 

 

 

Wall-E2 Motor Controller Study Part II

Posted 07 December 2019 (Pearl Harbor Day)

Back in May of this year I posted about different motor driver possibilities for my 2-motor and 4-motor robots, and showed some results for two drivers (an Adafruit Featherwing and an Adafruit DRV8871).  However, I got kind of sidetracked after this post when I discovered the RFI/EMI problem with the MPU6050 IMU.  At the time, I blamed the RFI/EMI problem on the non-linear nature of the newer drivers, and went back to using the L298N linear driver.

After quite a bit of experimentation and work, it finally turned out that most (if not all) the RFI/EMI problem was the Pololu 20D metal-geared motors themselves, and properly suppressing the noise at the motor terminals with bypass capacitors solved the problem.  So I have decided to repeat some of my initial motor driver testing.

Adafruit DRV8871 Single Channel Motor Driver Testing:

I removed the L298N driver I have been using up until now on my 2-motor robot, and replaced it with two DRV8871 Single Channel Motor Drivers. The hookup with the DRV8871’s is actually significantly simpler than the L298N, requiring only two control lines per channel instead of three.  After the normal number of errors, I got it running and started some long-term tests with the motors running continuously (with varying speeds and directions) while polling the MPU6050 every 200 mSec for yaw data.

This test ran for over 5 hours without problems, and then the GY-521 (generic MPU6050) module stopped responding to requests for data.  This is the second MPU6050 module that  has behaved this way – running for an indefinite amount of time and then refusing to respond.

The good news is, the DRV8871 motor drivers worked flawlessly the entire time, and are efficient enough so the T0-3 size chip container was just barely warm to the touch.

I have run  several long-term tests now with the DRV8871 drivers and two different MPU6050 modules, and the longest error-free run has been about 5 hours as shown above.  However, I have also had the test terminate in less than 5 minutes, so this is clearly not reliable enough for prime time. Also, adding my 2-stage power supply filter back into the system did not seem to effectively suppress errors, so no I have no clue what is going on.

11 December 2019 Update:

I ran the system overnight with the same test program, except I commented out the motor run commands so the motors themselves did not run. The test ran for over 11 hours and was still running fine when I terminated it.  So, the motors definitely have to be running for the problem to occur.

15 December 2019 Update:

Hoping to maybe eliminate a variable, I changed from the UNO controller to a Teensy 3.2. As usual, this ‘small change’ took a LOT more time than I thought it would.

There aren’t any good example programs for interfacing a Teensy 3.x with the MPU6050, especially using the I2CDEV/MPU6050/DMP libraries. I had a huge problem trying to track down compile issues with the I2CDev libraries, but in the end it came down to figuring out a way to get I2CDev to use i2c_t3.h instead of Wire.h. I solved this by copying I2CDev.h/cpp from my Libraries folder to my local project/solution folder, and editing the code to define the I2C implementation as I2CDEV_ARDUINO_WIRE and then replacing “#include <Wire.h>” with “#include <i2c_t3.h> in the appropriate section as shown below. I’m sure there are more elegant ways of doing this (maybe adding a ‘I2CDEV_TEENSY_3.X’ section at the top?)

After making the above change, the project started compiling OK, and I was able to connect to the MPU6050 and pull off yaw values using the DMP.

The next problem occurred when I tried to run a test program with the Adafruit DRV8871 drivers.  The two control lines alternate between being used as digital outputs (for direction control) and analog/PWM outputs (for speed control). Unfortunately, once a Teensy 3.2 line has been written to with ‘analogWrite()’, it can no longer be used as a digital output without first running ‘pinMode([pin],OUTPUT)’ on it. This particular little ‘gotcha’ appears to have been there since around 2016, but got lost in the shuffle as it is still there in the latest TeensyDuino libraries.

After fixing that problem, I was successful in both getting yaw values from the MPU6050 using Jeff’s I2CDev libraries, and in driving my Pololu D20 metal-geared motors via the Adafruit DRV8871 motor driver modules.

After getting everything working, I took the time to fork Jeff Rowberg’s I2CDev library, edit I2CDev.h/cpp appropriately and create a pull request for Jeff.  The changes were merged into the Github master distro pretty quickly, so in theory at least, all a new Teensy 3.2 user has to do is grab Jeff’s I2CDevLib stuff and go.

Here’s my Teensy 3.2 program for testing the MPU6050/DMP and the Adafruit DRV8871 motor drivers:

IMU Motor Noise Troubleshooting, Part II

Posted 13 November 2019,

After a month-long diversion to help Homer Creutz investigate some of the many issues associated with the Invensense MPU6050 IMU chip, I’m now back to figuring out how to keep my robots’ motor & driver noise from corrupting the yaw values from my MPU6050 IMU module.  In my last post on this issue, I worked through a number of theories for what might be causing the problem, and eventually decided the issue was more associated with the motor drivers than with the motors themselves.  At the conclusion of this prior study, I was able to demonstrate effective turn control using MPU6050 yaw values, even with the motors running.

However, I have since determined that I basically had gotten lucky; the problem still persists, even after changing back from switch-mode to linear motor drivers.  After a lot more work, I am now convinced that the basic problem is very high frequency voltage spikes on the order of 2-3 V p-p being conducted onto the Mega 2560 microcontroller board via the motor control connections from the Mega to the motor driver.  The result is intermittent and erratic behavior from the Mega and/or the MPU6050 module.

As a potential solution to the problem, I spent some time investigating whether or not I could use a ESP32 module to replace the Mega and it’s accompanying wireless module  (HC-05 BT module on the 2-motor robot, and a Wixel wireless serial extender on the 4-motor model). The idea was that since the ESP32 module is much smaller and has an integrated/shielded wireless module, it might be more immune to the conducted noise issue.  As it turned out though, getting the ESP32 to work the way I needed it to was next to impossible, and so I abandoned ship after a few weeks.

Anyway, now I’m back to working the motor noise problem again.  When I left the problem the last time, I thought that one possible solution to the noise conduction problem would be to use an optical isolator module such as this one, to isolate the motor power circuits from the Mega circuits.

So, I set up an 6-channel optical isolation bank between the Mega 2560 controller and the 2-motor robot’s motor controller, as shown below:

6-channel opto-isolator setup with 2-motor robot and Mega 2560 controller

However, when I tried this trick, I still got lots of high-frequency transients on the control and power lines, and the computed yaw values from the MPU6050 soon became invalid.  After poking around a bit with a scope, I realized that while this cheap 4-channel optical isolator is great for voltage difference isolation and low frequency isolation, the high-frequency stuff I was seeing was going around the optical isolation and capacitively coupling from the output side back to the controller side – bummer!

After getting back on the project, the first thing I did was to refresh my memory on a prior project to test a Sparkfun MPU9250 (basically the same as the MPU6050 but with a magnetometer included) interfaced to a Teensy 3.2 microcontroller.  The Teensy is much faster and physically much smaller than the Mega controller currently on my robots, so I thought it might be less susceptible to motor noise – worth a shot anyway.  So, I got the Teensy/MPU9250 configuration working again, and then did the absolute minimum to get the Teensy to drive just one motor on my 2-motor robot.  Surprisingly, the Teensy/MPU9250 combination went Tango Uniform as soon as the motor started up – wow!

So, now I wasn’t any closer to solving the problem, but I did have some additional information.  Now I knew that the problem wasn’t unique to the Mega 2560/MPU6050 combination – it also happened in the same manner with the Teensy 3.2/MPU9250. This indicated to me that the issue really was high-frequency noise spikes being conducted down the motor control wires and back into the microcontroller circuits.

So, I needed a way to effectively block these transients from reaching the microcontroller.  As noted above, I tried the cheap eBay optical isolator module, but although it clearly isolated the DC circuits, the high-frequency transients still made it through to the controller board. I needed an optical isolator setup with an ‘air gap’ big enough so that there would be no chance for the transient energy to jump the gap.  And, because to a man with a 3D printer, everything looks like a 3D fabrication problem, I figured I could whip something up using IR LED/IR phototransistor pairs, something like the following:

This model is just a small solid piece of plastic with 4 holes, sized so that a 3-mm LED / phototransistor will slide in, with a step to stop if from sliding all the way through.  There is no metal at all, other than the LED/phototransistor leads, so (hopefully) no conductive or capacitive path.

After the required three or four quick and dirty model iterations, I had a model that I could use for a very basic experiment.  In fact, I realized my little 4-hole model was way overkill, as all I really needed was one channel for the motor speed PWM signal – the other two inputs to the motor controller could be tied HIGH or LOW as necessary on the motor controller side for purposes of this experiment.

The setup is shown below:

Teensy 3.2 controlling one robot motor via homebrew optical isolator

Closeup comparing the 3D-printed optical isolator with the commercial 4-channel module

After getting the 1-channel optical isolator working, I used it to control one of the two motors on the 2-motor robot, and found that I could run this setup indefinitely with no data corruption – yay!  Here’s an Excel plot of a 35+ minute run with the robot motor running.  As can be seen from the plot, the MPU9250 responded properly the entire time. The five ‘spikes’ are caused by occasional manual sensor rotation to confirm proper operation

35-minute run showing reliable yaw data acquisition. The three ‘spikes’ are caused by manual sensor rotation to confirm proper operation

For the next step, I plan to expand my homebrew opto-isolator to 2 channels so I can control both robot motors.  If that is successful, I’ll add 4 more channels so I can control both motors completely.  If this all works, then I’ll need two complete 6-channel opto-isolators to control all 4 motors on the 4-motor robot.

15 November 2019 Update:

I added two more optical isolator channels so I could control both the speed and the direction of one motor on my two-motor robot.  Then I modified my Teensy 3.2 test program to run the motor at increasing and then decreasing speeds, both forward and backward, forever.  The motor speed increments and direction changes occur in the same section of code that acquires a new yaw value, every 200 mSec or so.  Here’s the setup

This setup puts about 30 mm separation between the two circuits, with the only connection being via my homebrew optoisolator.

I ran the system for over two hours with the motor running in both directions and with the speed varied across the entire range.  The motor, the program, and the yaw retrieval process worked perfectly the entire time, as shown in the following Excel plot and short video clip.

 

This clearly demonstrates that my homebrew optical isolator works as it should, and effectively isolates motor/driver related transients from the Teensy board.  What it doesn’t do is demonstrate the same capability with the Teensy connected to the same power supply; right now the Teensy is powered from my laptop via USB, and the motor is powered from the robot battery supply.  The next step will be to figure out how to suppress these transients to the point where both the Teensy and the motor/driver can live on the same circuit.

16 November 2019 Update

Success!  I now have the Teensy controller and the Robot running from the same power supply – yay!!  I accomplished this by constructing a two-stage RC filter between the 8.5V battery supply voltage and the 4-6V Teensy power input, as shown below, and modifying the USB cable to cut the red power wire.

Two-stage power supply filter for Teensy

2-stage power supply filter

With this setup, I ran for almost 30 minutes with no problems, as shown in the following Excel plot (the ‘glitches’ at the start and finish are manual sensor rotations to confirm valid data retrieval.

 

Stay Tuned!

17 November 2019 Update

According to this link, Thomas Huxley once said “The great tragedy of science—the slaying of a beautiful hypothesis by an ugly fact.”  Well, my ‘beautiful hypothesis’ has been well and truly slain by an ugly fact! As a last experiment to verify my hypothesis about the need for both an optical isolator and a power supply filter to adequately address the two-wheel robot’s motor noise transients, I bypassed the optical isolator as shown in the following photo, and ran the setup for several minutes.  Unfortunately (or fortunately depending on one’s point of view), the setup did not cooperate with my hypothesis – the setup ran fine, with no missed yaw value acquisitions, as shown in the following Excel plot (the ‘glitches’ in the plot are due to me manually rotating the sensor to verify proper operation).  So, apparently the power supply filter is required, but the optical isolator is not – Yay! (I think).

 

 

 

So then I removed the Female-Female jumpers and wired the motor control lines directly to the Teensy – no problem.  Then I added the three control lines for the second motor, and changed the Teensy program to control both motors.  This ran fine, so it is now clear that all that is required for motor transient suppression (at least for the Teensy setup) is a good power supply filter.  It’s even possible that my 2-stage power supply filter circuit is overkill for the application, and one stage would do fine.  To test this theory, I eliminated the first stage of the filter entirely, and tried again.  This works as well, so it looks like only a single-stage power supply filter is required for reliable operation, at least with the two motor robot and the Teensy microcontroller.  However, one fly in the ointment is that eliminating the first stage causes the power dissipation of the 5V Zener diode to increase to well over its nominal max Pd of 1W.  With a 10-ohm series resistor dropping 3.5V, the current through the zener is 3.5/10 = 0.25A (minus the negligible Teensy current), so Pd = 0.25 * 5 = 2.5W – oops!  I need to increase the series resistor by a factor of 2.5 to get Pd down to 1W.  So, I increased the series resistor to 20 ohms and with the actual power supply voltage of 8V, the zener Pd falls well below 1W.

For a final confirmation test, I eliminated the power supply filter entirely, bringing the setup back to to the original baseline, with the Teensy controlling both robot motors with no power supply filtering or optical isolator.  As expected, this caused the Teensy to lose synch with the IMU within a few minutes.

19 November 2019 Update:

After testing with the Teensy/MPU9250 combination, I decided to try and go back to the Arduino series of microcontrollers.  The reasons for doing this are:

  • The Teensy doesn’t have sufficient I/O channels to control all the required peripherals on the 4-wheel robot.  I does have enough I/O to control the 2-wheel robot, but that’s not what I’m after.
  • Remote programming of the Teensy is not possible without the use of another controller, like a Raspberry Pi Zero W or something like that.  However, this is easy to accomplish with the Arduino UNO or Mega 2560.

So, I started with an Arduino UNO that has been modified by removing T1 so board power is isolated from the USB cable power lead, and hooked up the motor control lines directly to the 2-wheel robot.  This arrangement failed after a few minutes, with or without power supply filtering.

So then I re-introduced my homebrew optical isolator, and found that the UNO will run indefinitely while running one of the two motors via the optical isolator and the power supply filter.  So at this point a reasonable hypothesis is that the Teensy is a bit more robust with respect to EMI/RFI effects than the UNO/Mega controllers, but the combination of the two-stage power filter and my homebrew optical isolator effectively suppresses motor and motor driver EMI/RFI (at least for one motor).

30 November 2019 Update:

I now have my two-motor robot running reliably with an Arduino UNO running my DF Robots MPU6050 module, with both motors running. As far as I have been able to determine, this requires both a power supply filter between Vbatt and the UNO Vin, and a 6-channel optoisolator between the UNO and the motor driver module.  Some photos and schematics are shown below:

6-channel homebrew optoisolator and power filter mounted on an Adafruit ‘Perma-Proto’ half-size breadboard

Optoisolator mounted on two-motor robot. Note DF Robots MPU6050 module in top left of photo

6-channel optoisolator schematic

 

03 December 2019 Update:  The Final Chapter (I hope)

After going through the entire process described above, proving to my satisfaction that the ‘final cure’ to the motor noise problem was to opto-isolate the motor driver control signals and thoroughly filter the power to the UNO, I was once again rudely smacked in the face by reality when my finely tuned setup refused to cooperate. As I prepared to start ‘field’ testing again, the yaw value corruption problem once again reared its ugly head.  This was beyond depressing – I have now spent upwards of 5 months trying to kill this particular alligator, and it keeps coming back.

Based on my philosophy that if I’m having an apparently insoluble problem, there’s someone (possibly many someones) out there in the i-verse that has had (and solved!) that particular problem already, I started over on internet research, googling for ‘motor noise problem’ and similar terms.  One of the hits I got was for an application note on the Pololu site dealing with just this problem (don’t know why I didn’t see it before, but…). In any case, Pololu’s solution was to install one or three bypass caps on the motor body itself.  Since I had already tried everything else, I thought ‘what the heck -it can’t get any worse than it is already’, and installed the three-capacitor arrangement, using non-polar 1 uF caps from each terminal to the motor body, and a 0.01 uF cap across the terminals.

Pololu D20 metal-geared motor with bypass caps installed. Blue caps are 1uF, orange cap is 0.01uF

Lo and Behold! It worked!  I was able to run for 90 minutes without a problem with all three elements in place; the motor bypass capacitors, the power supply filter, and the 6-channel optoisolator.  So, the next step was to bypass the optoisolator and see if that was a necessary component – and it continued to work without problems – yay!  Next, I bypassed the power supply filter circuit, and everything STILL continued to work great – double yay!

Two-motor robot with power supply filter and optoisolator bypassed

So, I wound up back at the beginning of my five-month circular journey, having learned a lot and had a lot of fun, but having wasted half a year.  So, if you are using metal-geared motors like the Pololu D20/D25 models, for Pete’s sake install bypass capacitors before doing anything else!

Stay tuned,

Frank

 

Sparkfun MPU9250 Test with Teensy 3.2

Posted 15 September 2019

After successfully demonstrating heading-based wall following with my little two motor robot, I attempted to integrate this capability back into my newly re-engined (re-motored??) 4-wheel Wall-E2 robot. Naturally the attempt was a dismal failure, for reasons I have yet to determine.  For some reason, the MPU 6050 IMU on the 4-wheel robot refused to produce valid heading data, and when I then attempted to redo the previously successful experiment on my 2-wheel robot, it too failed – in the same manner!  Clearly the two robots got together and decided to misbehave just to watch me tear what’s left of my hair out!

So, after trying in vain to figure out WTF with respect to the two robots and their respective IMU’s, I decided to just start all over with a different controller and a different IMU and see if I could just make something positive happen.  I found a Sparkfun MPU 9250 IMU breakout board in my parts bin, left over from an older post.  Because the Sparkfun board is set up for 3.3V only, I decided to use a Teensy controller instead of an Arduino Mega and see if I could just get something to work.

After the usual number of screwups and frustrations, I was finally able to get the Sparkfun MPU 9250 breakout board and the Teensy 3.2 talking to each other and to capture some valid heading (yaw) data from the MPU 9250.  The reason for this post is to document the setup and the code so when I have this same problem a year from now, I can come back here and have a working, documented baseline to start with.

The Hardware:

I used a Teensy 3.2 and a Sparkfun MPU 9250 IMU breakout board, both mounted on a small ASP solderless breadboard, as shown in the following photo, along with a Fritzing view of the layout

The Software:

I wrote a short program to display heading (yaw) values from the 9250, as shown below. The program uses Brian (nox771)’s wonderful i2c_t3 I2C library for the Teensy 3.x & LC controllers, and a modified version of ‘Class_MPU9250BasicAHRS_t3.h/cpp that incorporates an adaptation of Sebastian Madgwick’s “…efficient orientation filter” for 6D0F vs 9DOF.  The modification removes the magnetometer information from the calculations, as I already know that the magnetic field in my environment is corrupted with house wiring and is unreliable.

The modified Madgwick routine is included below

Note to self; after reviewing the extensive email thread with Kris Winer (tleracorp@gmail.com) I now believe I encapsulated all the required modifications to the AHRS code into a new class called “Class_MPU9250BasicAHRS_t3” with Class_MPU9250BasicAHRS_t3.h & .cpp files, and then referenced this new class in my MPU9250 work.

The results:

After getting everything working (and figuring out the history), I finally started getting reliable heading data from the MPU9250 as shown in the following Excel plot, where the breadboard was manually rotated back and forth.

Stay tuned,

 

Frank