Known defect in Arduino I2C code causes hangup problems

9 Replies

Posted 20 August 2018

06 July 2020 Update

Miracle of miracles!  Arduino finally got off their collective asses and decided to do something about the well-known, well-documented, and long-ignored I2C hangup bug.  Thanks to Grey Christoforo of Oxford, England for submitting the pull request that started the ball rolling.  See https://github.com/arduino/ArduinoCore-avr/pull/107 for all the gory details.  However, in a bizarre outcome, the implementation of the needed timeouts isn’t implemented by default! You have to modify your code to add a call to a new function, like the following:

Note that you have to explicitly add a timeout value (1000 in my example above) or the timeout feature will still not be enabled! The ‘true’ parameter tells the library to reset the I2C bus if a timeout is detected – surely something you will want to do.

I’m currently working on a ‘before/after’ post to demonstrate that the new timeout feature actually works with real hardware scenarios.  However, due to the intermittent nature of the I2C hangup bug, it takes a while (hours/days) to grind through enough iterations to excite the bug reliably, so it may be a while before I have a good demonstration

One last thing; at some point the examples in C:\Program Files (x86)\Arduino\hardware\arduino\avr\libraries\Wire\examples (on my Win 10 machine) will probably be updated/expanded to show how to properly implement the new timeout feature, but this has not happened yet AFAICT.

Stay tuned!

In my continuing quest to add relative heading sensing to Wall-E2, my autonomous wall-following robot, I have been trying to make the Invensense MPU-6050 module sold by DFRobots work on my robot.

In my last post on this topic, I had finally figured out that the program lockup problems I had been experiencing were due to a well-known-but-never-fixed bug in twi.c the low-level code associated with the Arduino I2C library.   This utility program has a number of while() loops used to send and receive bytes across the I2C bus, and every one of them is prone to deadlock when the device(s) on the other end of the bus misbehaves at all.   Then the while() loop never exits, and whatever program is running dies a horrible death.

The weird thing about this problem is that it has been known for at least a decade (yep – 10 years!!!), and has actually been fixed multiple times by multiple people over this period, but the fixes have never made it into the ‘official’ Arduino Wire library.   This makes  NO SENSE, as the Wire library code is open-source, and is available on GitHub.   I thought the whole idea behind open-source code and GitHub was that others could contribute code fixes in a reliable revision-tracked way, so that when someone finds a bug, it can be fixed quickly and then propagated out to all users.   Apparently the guys at Arduino never got the memo, because I found it impossible to get a ‘Pull Request’ containing the bug fix through the code-maintainer’s gauntlet.

Thinking this was just a logistics problem that I could solve with just a few hours of elbow grease, and would be a good training exercise for other open-source collaboration projects, I decided to take a swing at this problem myself – how hard could it be?

  • I thoroughly researched the technical issues, made the changes to my local copies of Wire.cpp/h and twi.c/h, and verified that they indeed fully solved the hangup problems
  • Found the releveant Arduino Wire library source tree on GitHub
  • Forked the Arduino Wire library source tree to my own GitHub Account
  • Cloned my fork of the Arduino Wire Library to my PC
  • Made all the relevant changes to my local repo, tested the result, and pushed the changes to my GitHub repo.
  • Created a ‘Pull Request’ with all the changes, with a descriptive note

By this time, I had expended a LOT of time, but that was OK as I had learned a lot that would pay off in future efforts, and besides I was finished – I thought!

Then I got a very nice email from the Arduino maintainer of the Wire library, listing all the things I had done wrong, and making it clear that the changes wouldn’t be merged into the ‘official’ Wire library until all was correct to their satisfaction.   When I looked at the list of problems, I realized most of it was about ‘whitespace’ mismatches between my submission and the official version.   Now, I don’t know about you, but I stopped thinking about whitespace a decade or so ago, when it became clear that whitespace was just a figment of the programmer’s mind, and had NOTHING WHATSOEVER to do with how well or poorly the code actually worked.   Now I was being asked to manually correct all the literally hundreds/thousands of places where my code had 2 spaces and the ‘official’ code had 3!   So, if I wanted this bugfix to get into the main distribution, I was going to   have to spend a HUGE amount of time dealing with nit-picking aesthetics that have nothing whatsoever to do with anything but somebody’s misplaced idea of right and wrong with respect to whitespace, for source files that are rarely, if ever, viewed by 99% of the Arduino programming community.   I mean, this would be like refusing to make a small, but important change to the maintenance manual for a car because the shop technician’s penmanship wasn’t up to par!   What is penmanship going to matter when known defects aren’t corrected?

So, I thought about that some more, and I came to realize why this I2C hangup bug has been around for so long – nobody’s pull request has ever made it through the ‘penmanship contest’ gauntlet; the Arduino maintainers are more interested in penmanship than in fixing clearly defective code that has (and still is) causing grief for anyone who tries to use the I2C bus.   My personal response to this problem was “screw them – I’m not going to spend all that effort just to please someone’s weird affection for whitespace, especially since my local copy of these files has already been fixed.

With just a little bit of searching, I found Steve Bian’s ‘SBWire’ library with timeouts added to all the while() loops in twi.c, and was quickly able to ascertain that Steve’s library did indeed solve my hangup problems.   Moreover, Steve actually answered my emails, and is undoubtedly much more open to open-source collaboration than the guys at Arduino.

The sad thing about all this is that Arduino is not doing themselves any favors by making themselves part of the problem rather than part of the solution. If they aren’t going to actively maintain their baseline code distribution, it (and Arduino) will become irrelevant as users find other ways around the obstacles.

Frank

25 August update:

So, I did the same thing with Shuning (Steve) Bain’s SBWire library that I had done with Arduino’s Wire library.   Forked his repo, cloned it to my PC, made the small changes I wanted, pushed to my repo, and created a pull request.    Two Days later, Shuning had merged my changes into the library.   Now I do realize that SBWire isn’t ARduino Wire, so maybe a ‘higher standard’ might be justified for the ‘gold standard’ I2C library.   However, I think we could all agree that EIGHT FRIGGIN’ YEARS  of known defects is probably a bit much!

So, my advice, if you’ve been having problems with I2C hangups, is to throw the Arduino Wire library in the nearest trashcan and use Shuning’s SBWire library

26 August Update:

I have been running SBWire on a little I2C test board, and I left it running over the weekend while my wife and I were away on a trip.   When I came back, some 95 hours later, the board was still running merrily.   I did note that the ‘lockup counter’ (the number of times the standard Wire library code would have locked up) stood at 14, or about once every 7 hours or so.   Actually I’m a bit surprised by this number, as in my personal experience the Wire library never lasted more than about 2 hours before locking up.

Just another reason to dump the Arduino Wire library and use something useful like SBWire ;-).

Integrating Time, Memory, and Heading Capability, Part V

9 Replies

Posted 10 August 2018

Well, it appears I spoke too soon about having solved the I2C hangup problem on my Wall-E2 wall-following robot.   In my last post on this subject, I described all the troubleshooting efforts I employed to nail down the cause of intermittent hangups when trying to use the MPU6050 6DOF IMU on the robot, along with several other I2C devices (a Teensy 3.5 used for IR homing, and Adafruit RTC, and FRAM modules).

After (I thought) figuring out that the I2C SCL/SDA line lengths were the root problem of the hangups I had been experiencing, my grandson Danny and I spent some quality time reworking Wall-E2’s layout to accommodate shorter line lengths.   Instead of mounting the IMU and it’s companion sensors on the second deck as before, we 3D printed a small plastic plate to attach to one of the hexagonal 2nd deck standoff posts and provide a 1st deck mounting area for the sensors.   The previous and new mounting locations are shown below:

2nd deck mounting location. The MPU6050 is the module with the illuminated blue LED toward the rear of the robot

1st deck mounting location for I2C sensors (lower right-hand corner of the photo). The Teensy 3.5 IR homing module is shown mounted on the IR detector housing (above the red plastic plate)

Unfortunately, as I was doing some final tests on this setup, I started experiencing hangups again.   After a day or so moping and some very choice words, I started all over again trying to figure out what happened.

On previous searches through the i-verse, I had run across several posts indicating that the Arduino Wire library had some basic problems with I2C bus edge conditions; there were several places where it uses several blocking ‘while()’ loops to transmit and receive data on the I2C bus, and there was no way to recover from a ‘while()’ loop where the exit condition was never satisfied.    After literally exhausting all the other possibilities, it was becoming apparent that this must be what was happening – the MPU6050 must occasionally fail to respond correctly to a I2C transaction, causing the associated ‘while()’ loop to never exit.

So, I started looking for solutions to this problem.   Again, I found some posts where folks had modified the low-level I2C bus handling code found in twi.c/.h, the code underlying the Android Wire class.   I found a post by ‘unaie’ (http://forum.arduino.cc/index.php/topic,19624.0.html) with the same complaint, but he also posted modified versions of twi.c and twi.h that solved these problems by forcing the ‘while()’ loops to exit after a set number of iterations, and resetting the I2C bus when this happens.   His modified versions can be downloaded at:

http://liken.otsoa.net/pub/ntwi/twi.h

http://liken.otsoa.net/pub/ntwi/twi.c

I downloaded these files and tried to replace the ‘stock’ twi.c/h with the modified versions. Unfortunately, unaie’s modifications were made on a quite old version of the files, and conflicted with the later ‘repeated start’ versions of these files that are in the current ‘wire’ library.

So, I did a ‘diff’ between the ‘repeated start’ version and unaie’s version, and created a modified version of the latest ‘repeated start’ twi.c/h.   In addition, I added a couple of functions to allow monitoring of the number of times a bus reset was required due to a ‘while()’ loop timeout.   When I was finished, I ran the sensor for over 24 hours with no failures, but in that time there were three instances where a ‘while()’ loop timed out and a I2C bus reset was required.   A small snippet of this run is shown below.   The blue line is the yaw value, and the plot snippet shows where I manually rotated the sensor just after 24 hours, and the horizontal orange line shows the number of bus resets.

Small snippet of 24-hour sensor run. blue line is reported yaw value; orange shows the I2C bus reset counter

So it is clear that, absent the lockup recovery modifications, the I2C bus would have locked up long before, and that with the modifications ‘while()’ loop deadlocks have been successfully handled.

11 August 2018 Update:

The sensor is still going strong after 44 hours with no hangups, and the reset counter is still holding at 3.

The complete twi.c & twi.h codes are included below:

 

Stay tuned!

Frank

 

Integrating Time, Memory, and Heading Capability, Part IV

1 Reply

Posted 07/26/18,

In my last post on this subject, I described my efforts to troubleshoot an intermittent ‘bad data’ problem I experienced with the Inversense MPU6050/DMP breakout board from DFRobots.   The calculated yaw value would occasionally ‘lose synch with reality’ and start varying wildly.   I finally tracked this down to occasional bad reads from the DMP FIFO; the read would extract bytes from two different packets, producing an invalid yaw value.   The fix for this problem is to monitor the FIFO ‘bytes remaining’ count and reset the FIFO whenever the ‘bytes remaining’ count is not an integral multiple of the packet size.

After demonstrating that I could run the MPU6050 for days (literally) without any bad data occurrences, I thought I was home free in my effort to provide relative heading information for precise turns, but I ran into yet another problem when I tried to integrate the MPU6050 back onto the robot.   This time the problem was a hangup problem; after some tens of minutes, the program would stop responding at all – as if it had gone into an infinite loop somewhere (which is exactly what was happening, but I’m getting ahead of myself…).

This problem turned out to be a ‘simple’ I2C wire-length issue; I was able to demonstrate that the problem would occur any time the I2C ICL/SDA wire length went beyond about 42 cm, and would not occur with wire lengths below about 30 cm.   The problem did not appear to be sensitive to I2C bus speed (at least not for the default 100KHz or for a reduced clock speed of 50KHz) or pullup resistor value – just to wire length.

The rest of this post is a copy of my troubleshooting notes from the effort to track this problem down and solve it.   I have found in the past that when facing a non-trivial problem with multiple possible causal factors, a troubleshooting journal is an absolute must.   In the pre-computer days (you do remember there was a time before ubiquitous computing don’t you?), I used a real MIT engineering notebook for this purpose, and then later on a 3-ring binder into which I added quadrille-ruled sheets covered with my notes and drawings.   Then I moved on to Word documents – nicer because I could include Excel plots, Visio drawings, photos, and other mixed media.   Now that I have graduated to a WordPress blog, I can use it as a repository of my working notes, while also allowing others to see the inner workings of a mad scientist’s mind ;-).

I2C Hangup problem with Inversense MPU6050

Download

Stay tuned,

Frank

 

Integrating Time, Memory, and Heading Capability, Part III

Leave a reply

Posted 12 July 2018

In the last installment of this particular saga, I described another chapter in my ongoing effort to add heading knowledge to Wall-E2’s (my autonomous wall-following robot) super powers.   In that post, I described my attempt to utilize the Inversense IMU 6050 6DOF breakout board from DFRobots.   I posted some results that showed problems with vibration screwing up the results, and then getting error-free results using an ‘air pillow’ (a piece of air-filled packing material).   At the time, this led me to believe that the cause of the bad data was motor vibration.   However, when I tried adding some foam vibration dampening material, it didn’t seem to help – I was still getting intermittent stretches of bad data, with or without the motors running.   Clearly I  still didn’t understand what was happening.

Once again I ran away from the whole thing, to regroup and let my mind work on the problem for a while; back to basketball, bridge, and general goofing off.

After some more web research and just thinking about what I knew and didn’t know, I started to suspect that what I was seeing was an artifact of the way the sensor(s) communicated with the main controller via the I2C serial interface.   When yaw measurements went bad, they went  really bad, rapidly cycling from positive to negative values, and this didn’t make a lot of sense.   Maybe the main controller wasn’t keeping up with the sensor data stream, and   the software was trying to form heading values using bits from two different measurements; this would explain why the heading sign changed from measurement to measurement.   Also, I wasn’t utilizing the INT pin on the IMU6050 module, just pulling data out of the FIFO as rapidly as possible; could that be part of the problem too?

So, I decided to start all over again with the IMU6050 sensor on an ASP plugboard, with a spare Arduino Mega 2560 controller identical to the one being used to run Wall-E2, as shown in the following photo.   I also hooked up the INT pin, and used Jeff Rowberg’s I2CDev materials and MPU6050 example programs as the starting point.

DFRobots Inversense IMU6050 breakout board (board with blue LED, between FRAM and RTC) on an ASP plugboard, controlled by an Arduino Mega 2560

After getting everything going, I ran some long-term tests to see I could produce ‘bad’ yaw readings independent of the robot platform.   And, of course, I couldn’t get the darned thing to fail, no matter how long I ran it.   Shown below is a 20-minute long plot

20-minute run with no observed problems

Next, I tried inserting some delays into the ‘do other stuff’ part of the main loop, to try and simulate the normal robot processing delays.   This had no effect up until the delay reached 40mSec or so, and then I started to see problems very similar to what I had seen before with both the MPU9250 and 6050 sensor setups.

On robot test displaying yaw value and bytes remaining in MPU6050 FIFO

Then I modified the code again to check for FIFO byte lengths that weren’t an integral multiple of the normal packet length (42 in this case), and to reset the FIFO if the condition was detected. This seemed to eliminate the ‘bad data’ effect, regardless of the amount of delay introduced in the processing portion of loop().

About 6 minutes with the motors running. Program modified to reset the FIFO whenever a non-modulo ‘bytes remaining’ condition was detected

Detail view of the last 100 seconds of the previous plot

Summary:

The Invensense MPU6050/DMP/FIFO combination is sensitive to delays in the main   processing loop, even when using the INT line with an Interrupt Service Routine (ISR).   When the main loop processing delays get beyond about 40mSec, the  ‘mpu.getFIFOBytes(fifoBuffer, packetSize);’ call will occasionally  not remove the correct number of bytes from the FIFO, leaving a non-modulo (packetsize) number of bytes remaining in the FIFO.   When this happens, the next read will get (and process) bytes from two different packets, resulting in wildly varying yaw value outputs.   This condition is (now, after knowing what’s going on) fairly easy to recognize, as the error condition generally causes adjacent yaw values to have different signs, resulting in a classic sawtooth output.

The way to eliminate this artifact is to check for non-modulo (packetsize) FIFO bytes remaining value each time, and reset the FIFO when this happens.   Whatever good data is still in the FIFO will be lost, but the data that you do get will be valid.

I have included below my test program, with the FIFO modulo check and FIFO reset mechanism.   Note that this program also includes my motor control code, which obviously will not work with your setup.

 

Stay tuned,

Frank

 

 

 

 

Integrating Time, Memory, and Heading Capability, Part II

1 Reply

Posted 06 June 2018

About a month ago in early May I started another try at adding time, memory, and heading super-powers to Wall-E2’s repertoire.   In this post, I described the effort to add a FRAM board, an RTC board, and a 9DOF MPU9050 IMU board to Wall-E2’s already-existing I2C bus, and in this post I described fabricating a permanent ‘tri-sensor board’ for installation on Wall-E2’s second deck.

After installing the tri-sensor board on the second deck, I did some tests to confirm Wall-E2’s new found ability to make accurate turns, and instead of confirming success, I witnessed complete failure – Wall-E2 was even worse at making turns now than in the past when he was just doing timed turns – WTF!!

Backing up and doing some more tests, it quickly became apparent that the MPU9050 IMU worked fine until Wall-E2’s wheel motors started running, but went completely ga-ga after that.   I tried some quick tests to try and isolate the problem, but didn’t get anywhere.   So, I did what I always do when faced with an apparently intractable problem – I ran away ;-).   OK, I didn’t  really run away, but I decided to put the whole thing on the back burner for a while and let my subconscious work on it for a while (I have discovered over and over again that my subconscious is a  lot smarter than I am).   In the meantime I   had some other stuff to do, like going to a duplicate bridge tournament and going to a grand-daughter’s HS graduation.

After doing some web research, I began to suspect that the problem might be associated with the complexity associated with the Sparkfun 9DOF MPU9050 chip and the associated yaw/pitch/roll computation software, so I decided to see if there were other IMU solutions out there.   I ran across the DFRobots MPU-6050 6DOF IMU breakout board.

Since I wasn’t planning to use this for anything but yaw (z-axis rotation) information, the 6DOF limitation wasn’t a problem, and the integrated  Digital Motion Processorâ„¢ (DMP) should simplify the control software.   In addition, the IMU6050 board is 5V compatible, so I can eliminate the level-shifting hardware required for the Sparkfun 9250 unit.   So, I ordered a couple of units and started playing with them.   Using the supplied libraries from DFRobots and their little example program,   I was able to get the unit up and running fairly quickly.   To test the ability to report yaw excursions, I modified the test program to report only yaw, and manually turned the sensor around its Z-axis.   As shown below, the yaw progressed smoothly.

Manually turned sensor while recording yaw

Next, I mounted just the IMU6050 board on the robot’s 2nd deck, as shown below, and ran some tests with and without the motor running.

MPU6050 mounted on robot with double-sided tape

Sensor mounted on robot with double-sided tape, no motors running

As shown above, with the sensor mounted on the 2nd deck and with motors  not running, I got a smooth response from the yaw sensor as the robot was manually rotated.   However, when I tried the same trick with the motors running, I got the following plot:

Sensor mounted on robot with double-sided tape, left rear motor running

Then I mounted the sensor on an ‘air pillow’ (one section of a piece of air-filled packing material) and tried again, once again getting a smooth plot, even with one motor running.

MPU 6050 mounted on robot with an ‘air pillow’ (piece of some air-filled packing material)

6050 sensor mounted on air pillow, left rear wheel motor running

 

Integrating Time, Memory, and Heading Capability

2 Replies

Posted 06 May 2018

For the last two months I have been working on adding some secondary, but still important capabilities to Wall-E2, my wall-following autonomous robot.   As I noted back in April, Wall-E2 still can’t tell which way he is heading, which (among other problems) means he can’t make accurate turns.   In addition, Wall-E2 can’t tell how long (or even  if) he has been turned off, making it impossible to tell how long it has been since he last was charged.

In the intervening months, I have been able to obtain and test individual modules to address the above issues; the Sparkfun MPU9250 9DOF IMU breakout board to obtain heading information, and the combination of an Adafruit MB85RC256V FRAM breakout board and an Adafruit DS3221 RTC breakout board to capture the date/time of any power interruptions.

Since all three of the above breakout boards are I2C-capable, it should be feasible to run all three from a single I2C bus on Wall-E2’s main controller – an Arduino Mega 2560.   Unfortunately, the Sparkfun MPU9250 module isn’t 5V tolerant, so integrating all three isn’t as simple as just daisy-chaining them all together.   Back to Google for some more research, where I eventually uncovered Philips Application Note AN97055 by Herman Schutte of The Netherlands dealing with bidirectional level-shifters for just this problem. Turns out that Philips introduced the I2C bus back in 1980, so they may know a thing or two about the issues ;-).   In any case, this 1997 paper described a 2-MOSFET bidirectional level-shifter that completely addresses the issue, and seemed to be pretty straightforward to implement.   It took me a couple of tries, but I got it working, with the result that I can now run all three modules (RTC, FRAM and IMU) from an Arduino Mega 2560, with the level-shifting MOSFET’s placed between the IMU and the rest of the circuit, as shown in the photo below.

All three sensors operating a the same time, controlled over a single I2C channel.

In the above photo, the modules are (from left to right): Adafruit MB85RC256K FRAM breakout, Adafruit DS3231 RTC breakout, and Sparkfun MPU9250 9DOF IMU breakout.   The two  2N7000 level-shifter MOSFETs can be seen at the top left-hand corner of the Sparkfun IMU breakout board.

I put together a small test program, combining pieces from the software used to test the power-down date/time capture idea, and the software used to test the ability to use the Sparkfun IMU to accurately manage rotations.   This program doesn’t do much at all, but it does demonstrate that all three modules can be controlled at the same time over a single I2C channel, as shown in the following output.

As shown in the above output, all three modules are initialized, and then the program enters a loop where the current time is displayed from the RTC, and the current IMU heading is displayed from the IMU.   Then every 10 times through the loop, a new value is written to the FRAM until the first 20 or so locations have been written. Then the program starts ‘un-writing’ the values until all the written locations have been cleared, at which point the cycle starts all over again.

So, now that I have conclusively demonstrated the ability to add (relative) heading and FRAM-based non-volatile power-cycle date/time recording to Wall-E2, the next trick will be to actually mount the modules on the robot.   This will entail solving yet another set of problems, as it turns out (naturally) that although I have plenty of room on Wall-E2’s second deck, I have run out of available pins on the inter-deck connector.   This could be addressed by putting the modules somewhere on the first deck, but finding that ‘somewhere’ is going to be a real trick.   Time for either a second inter-deck connector, or to replace the current 8-pin model with a larger one.

10 May 2018 Update:

After verifying the proper operation of all three modules one one of my new high-quality ASP protoboards, I transferred everything to a permanent perfboard rendition more suitable for mounting on Wall-E2’s second deck, as shown below

Stay tuned!

Frank

 

 

Another try at heading information for Wall-E2

2 Replies

Posted 24 April 2018

A little over two years ago I started a project to give Wall-E2 ‘a sense of direction‘ by integrating a Mongoose 9DOF IMU board onto the robot chassis.   I worked on the idea for about six months before I finally determined that the magnetometer idea was not going to work in my indoor environment, as there was just too much magnetic interference due to indoor wiring, air handling motors, and the like.

However, it recently occurred to me that although the absolute magnetic heading problem was   intractable, I might be able to use one of the newer single-board IMU products (like the Sparkfun 9250) to generate  relative heading information, which I could use to solve a different problem.   Wall-E2 occasionally needs to perform sharp turns, on the order of 90 º, either as part of what I call an ‘open room step turn’ (like what happens when Wall-E2 exits a hallway into an open room, and needs to turn 90 º in one direction or the other to continue wall-following), or as part of an evasion maneuver or as part of a recovery from a stuck condition.   In my ‘field’ testing to date I have noticed that Wall-E2’s effective turn rate varies considerably depending on the surface condition; very slow on shag carpeting, OK on tightly-woven rugs, and very fast on hard flooring.   This means that my current strategy of timed turns only works OK on the medium surfaces, but sucks the big weenie on the other two types.   So, if I had even relative heading information, I could use it to make sure Wall-E2’s  90 º turns are actually  90 º, as opposed to  45 º or  180 º.

It turns out there are a  lot of single-board IMU solutions out there new, probably due to the wildly popular quad-copter market; they all need attitude-control (AHRS) module of some kind to make them flyable, and those modules need to be small and lightweight.   The one I started playing with first is the Sparkfun MPU-9250 breakout board, with 3-axis accelerometer, gyro, and magnetometer capability in a really small package for an incredibly cheap price.   And, as a bonus, there is a lot of good Arduino/Teensyduino software out there to help us mere mortals run the thing.   In particular, Kris Winer (of Pesky Products fame) has done a lot of work with sensor fusion software for the Arduino and Teensy line, and actively supports both his products and his software.

So, I got a board from Sparkfun and started playing around with it.   The first problem I ran into is that the MPU9250 is a 3.3V board, so I couldn’t run it from an Arduino without either potentially frying the board or having to implement a bidirectional level shifter. So, I grabbed a Teensy 3.2 from my stash and used it instead.   This, in turn, required some judicious sifting through the available software to find the Teensy 3.x compatible versions, but I eventually got that accomplished and started collecting data.

As usual, there were a lot of mis-steps along the way.   The biggest obstacle was figuring out how  to  avoid using the magnetometer data; almost all the software, including the all-important Madgwick quaternion routine for converting raw sensor data into yaw/pitch/roll values assumes the use of magnetometer data to obtain geo-magnetically referenced heading and to minimize/eliminate gyro drift.   In my case, I specifically wanted to decouple heading calculations from magnetometer input, as I already knew the magnetic environment in my house was too variable for reliable measurements.   With some guidance from Kris, I eventually found a 6DOF (3D rate gyro and 3D accelerometer) version of the Madgwick routine.

Once I started getting reliable yaw (heading) measurements from the sensor, I decided to modify my Teensy-based stepper motor rotary table measurement system to rotate the sensor in a uniform way to obtain constant-rate heading scan data.   The basic idea of the rotary table system is to rotate the unit under test X degrees, then stop, take some measurements, and then repeat.   So, I got it all set up, and got the following plot.

This data made no sense at all – for a full 360 º rotation I should have recorded a full 360 º heading change, regardless of the rate of rotation.   Instead, the sensor seemed numb to rotation rates below about 20 rpm (120dps) and even at 60 rpm (360 dps) I wasn’t seeing a full 360 º rotation – what the heck??

After beating on this problem for waaaayyy too long, it finally occurred to me that my test program had a fatal flaw; my measurement system stepped to each desired heading, then stopped, took a heading measurement, and then moved on to the next heading.   The operative word here being  stopped.   While this works great for units under test whose performance varies with rotation, the Sparkfun sensor performance varies with rotation  rate, not the rotation position!   Well, doh – it’s an accelerometer!   So, my carefully assembled test setup wasn’t measuring the sensor’s rotation rate at all – the rotation rate was being reduced to almost zero at each measurement position – oops!

After this “aha!” (or maybe, “doh!”) moment, I realized I needed to modify my rotary test system to make sure that heading data acquisition from the sensor was accomplished ‘on the fly’.   Once I did this, I started getting more reasonable data, like the following plot:

Nice, linear data – great!

At this point I started thinking about how I could integrate this capability into my robot to monitor/manage turn operations, and thought that I could maybe average the first few heading values before the start of a turn to create a stable reference to be used to terminate the turn appropriately.   However, when I tried this trick, with a 10-sample average at the start, I got the following plot

I performed the ‘with’ and ‘without’ initial measurement average experiments several times, and convinced myself that the above phenomenon was real, but I still have no real idea why it occurs.   It has to have something to due with the Madgwick quaternion manipulations, as that is the only place in the entire system where there is any state memory (the 4-element quaternion array itself).   For my purposes, it was sufficient to realize I couldn’t do what I wanted to do, and to “run away!” from this idea.

Next, I modified my stepper motor rotator test system again to simulate the process of setting up a turn direction and angle and terminating the turn when the correct turn angle change had been achieved.   Here’s a short video of the result, where the sensor is turned through +/- 30, 60, 90, and 180 º angles.   The motor is run at a constant rate until the target angle is approached, and then run at 1/5 normal speed to fine-tune turn termination.

So, at this point I confident that I can use the Sparkfun MPU9250 9DOF sensor (used in 6DOF mode) to accurately turn my robot – cool!

With the addition of the Sparkfun accelerometer and the FRAM/RTC combination to his sensor suite, Wall-E2 is set to get significantly smarter; he’ll be able to remember when he was last turned off so he can more accurately report run times and charge times, and he’ll be able to make real 90 º turns instead of having to fake it with timing.   Now if I could only get him to take out the trash! ;-).

Stay tuned,

Frank

 

 

 

Capture Power Loss Data Using Adafruit FRAM Breakout Board

2 Replies

Posted 06 April 2018,

As an enhancement to Wall-E2, my wall-following robot I wanted a way to capture the run time at power-down so I could determine how long it has been since Wall-E2 last charged its battery, even through power cycles.

At first I tried to do this using the Arduino Mega 2560’s  onboard EEPROM, but this proved infeasible, as the time required to write to EEPROM exceeded the time available from the time power was removed to the time the Mega died.   I played around with extra capacitance on the power bus, but this still wasn’t enough to hold the power up long enough to write to EEPROM.   And, even if I finally succeeded, there was still the problem of EEPROM wear-out to deal with.

So, after some more research time with Google, I found this nice Adafruit FRAM breakout board, featuring the Fujitsu M85RC256V 32KB FRAM part

Adafruit provides a simple, but effective I2C interface library and an example sketch, and I was able to use these with a spare Arduino Uno board to verify that I could indeed write to and read from the FRAM.   However, what I really wanted to determine was whether or not the FRAM was fast enough to allow me to write data to it after power was removed from the Arduino but before the processor actually died.

I did some poking around in the Arduinio Uno schematic and determined there was a 47μF capacitor on the output of the +5V voltage regulator, and so I thought I might be able to trigger off power loss at the input to the regulator and accomplish the writes while the capacitor was still holding up the Arduino processor.   I put a 1/3:2/3 voltage divider on the +12V input line and used my trusty Tektronix 2236 scope to monitor it and the +5V regulated output.   With the scope I was able to see that the +5V output stayed up for about 5msec, and stayed above 4V for 8msec, as shown in the following scope photo.

Scope photo of Arduino Uno +5V regulated output, triggered by removal of +12V input power. Time scale 1msec/div.

The next step was to modify Adafruit’s FRAM read/write example program to implement a power-down interrupt routine to test my idea.   Here’s the code (the software is also available here)

The above code takes advantage of a modified version of Adafruit’s I2C FRAM library that facilitates writing and reading of arbitrary data types like int, long, float, etc.   The modifications were cribbed from  Nick Gammon’s wonderful ‘IC2_Anything’ library – thanks Nick!

Here’s the modified ‘Adafruit_FRAM_I2C.h’ file

With this test setup, I was able to repeatedly clear the FRAM memory and see the effect of pulling the power plug.   A typical result is shown below:

Program response when restarted with CLEAR_FRAM_PIN (pin 3) grounded

Program response after the power plug was pulled at about 8.9 sec after startup.

In the first figure above, the  CLEAR_FRAM_PIN (pin 3) was held LOW while the Arduino was restarted, resulting in zeroes being written to the first 100 FRAM locations.   In the second figure, the +12V power plug was pulled after about 9 seconds, then reconnected.   When the Arduino restarted, the power-down ISR had written program timer values from 8976 to 8990 into the first nine 4-byte segments.   This shows that the Arduino continued to operate for about 14msec after the power plug was removed.   This is very good news, as it implies that I should be able to write the current date/time value from a real-time clock (when I get it running, that is) into FRAM whenever there is a power interruption, allowing me to accurately track battery usage history.

The hardware used to perform these tests is shown in the following photos:

Experimental setup moved to new ASP solderless breadboard

Arduino Board Modification:

I ran into a problem when I started testing the power-down interrupt idea; I wanted to keep thE Arduino board connected to my PC via the USB cable, but if I did that, the Arduino would automatically switch over to USB power when I disconnected the +12V power cable. Fortunately, if you have a problem, it’s almost certain that someone else has had and solved the same problem.   In my case I found this post, that explained that T1, the automatic power crossover MOSFET switch had to be removed, as shown in the following photo.   This modification allows the USB cable to continue to supply USBVCC power to U3, the ATMEGA8U2-MU chip, which in turn allows the PC to recognize the Arduino for firmware uploads.

Loader Loading...
EAD Logo Taking too long?

Reload Reload document
| Open Open in new tab

Download

So, now that I have demonstrated the practicality of a power-down interrupt routine to capture time-of-shutdown information (at least on an Arduino Uno), the next step is to integrate this capability with the Adafruit RS3231 Precision RTC breakout board.   Stay tuned!

09 April Update:   Adafruit DS3231 RTC breakout board added:

As I mentioned in my ‘Time and Memory for Wall-E2‘ post, I planned to complement the FRAM capability with the addition of an RTC module so that I could record the time & date of any power interruptions.   As I mentioned in that post, I planned to use the Adafruit DS3231 Breakout Board for this purpose.   After receiving the module, I added it to the system by daisy-chaining the I2C SCL & SDA lines from the FRAM module, and modified the software to incorporate the RTC.   I changed the FRAM_ISR() function to acquire ‘Unixtime’ from the RTC (i.e. the number of seconds since 00:00:00 UTC on January 1, 1970) and write it to the FRAM.

I tested this by uploading this firmware to the Arduino Uno, and then pulling the power plug a few seconds after the program entered the main loop.   Here’s the printout from the run

Program run with Adafruit DS3231 RTC added. Note ‘unixtime’ printout at top, and first 5 FRAM locations at bottom

As can be seen from the above printout, the power-down ISR successfully recorded the current time (in ‘Unixtime’ format) into five successive FRAM memory locations before the Arduino died.   Note also that the recorded time was eleven seconds after the time shown at the top of the printout – about right, as the upper value is the time retrieved using the PC’s __DATE__ and ___TIME___ environment variables at the time the program was compiled and uploaded to the Arduino board.

Here are some photos of the new hardware setup:

Adafruit FRAM module on the left, DS3231 RTC module on the right

Frank

 

Time and Memory for Wall-E2

2 Replies

Posted 01 April 2018 (not an April Fool’s joke)

Now that I have Wall-E2’s charging module and charging station working, I’ve moved on to some secondary, but still important, issues that need addressing.

  • Wall-E2 still can’t tell which way he is heading.   This isn’t necessarily a killer, as his current technique of simply following walls does fairly well.   However, when it comes to a human figuring out where Wall-E2 is or has been, distance to the nearest wall just doesn’t cut it.   A while ago I attempted to solve this problem using an onboard magnetometer, but ultimately concluded a magnetometer-based navigation system is doomed to failure in an indoor environment, due to too many interfering magnetic signals.   I have decided to try this again, but this time using a cheap solid-state gyroscope module from Sparkfun.    I should be able to initialize the gyroscope z-axis to an absolute heading each time Wall-E2 connects to a charging station, as their location(s) and headings are known.   As a bonus, I should be able to use the gyro to generate accurate 90 º turns, instead of the current open-loop timing method.
  • Wall-E2 remembers how long it has been since its last recharge, but only if the main power hasn’t been interrupted.   If I turn him off for any reason, that information goes away.   I dicked around with writing the current value of millis() to EEPROM on power-down, but it turns out that EEPROM writes are  way too slow for that.   After Googling around for a bit, I found a nice little FRAM (Ferro-magnetic Random Access Memory) module from Adafruit, and I believe I will be able to implement a power-down memory save feature using it.
  • Once I have a fast non-volatile FRAM solution, it occurs to me that I may want to write telemetry information to it, so it isn’t lost when Wall-E2 is out of range of the current Wixel link to my PC.   Maybe even set the FRAM (or at least part of it) up as a rate buffer between Wall-E2 and the Wixel.   The idea would be that telemetry data always goes to the FRAM at some rate A, and is then read from the FRAM to my PC via the Wixel link at rate B, where B > A.   When the link isn’t available, the telemetry data continues to be written into the FRAM, and is read back out again when the link becomes available.   As long as the link isn’t interrupted for too long, I won’t lose any telemetry data.   As part of this implementation, I would like to time-stamp the data with real date-time information, which requires a battery-backed RTC (real-time clock).   As it happens, Adafruit has one of these too, so I may be able to implement it quickly and easily.

I chose the I2C versions of all these modules, as I already have I2C implemented for acquiring steering cues from my IR Homing Module.   In theory, at least, adding three more I2C slave devices to an already existing setup should be trivial.

Stay tuned!

Frank

 

Charging Station Voltage Change From +5 to +12V

Leave a reply

Posted 22 March 2018

With the replacement of my Power Boost 1000C – based charger module with the TP5100, I needed to change the charging station supply voltage from +5V to +12V.   Unfortunately, the modulated IR beam signal is generated by a Teensy 3.2 module, which requires +5V (it’s actually a 3.3V module, but can accept power of up to +5V), so now I needed both +5 and +12V on the charging station.   The answer was to add a simple 3-pin regulator, as shown in the schematic below

Updated charging station schematic showing addition of a 3-pin 12-to-5V regulator

The original Teensy 3.2 side

The original 5V charging station layout, rear view

The new +12 to +5V regulator side

Updated charging station assembly, rear view