It was in September of 2001 that I was on leave-without-pay from my job at Northrop-Grumman, my wife was on doctor-ordered bed rest because our 3rd son was trying to “arrive early” (and we had two other toddler sons, ages 4 and 2), and I was working 12+ hours a day in my garage on a special personal project. The project was focused on demonstrating that a Neural Network attitude control system could successfully stabilize a radio-control helicopter along the roll and pitch axes. I’d created many successful Neural Network applications in simulation in the past but this was the first time that I was attempting to implement a Neural Network solution for a complex hardware-driven, real-world system.
The basic objective was that, given a set of commanded values for roll and pitch attitude, the helicopter Neural control system would solidly maintain the helicopter at those attitudes – thus it should be able to hover in a stable manner (if the commanded roll and pitch angles were appropriate for hover). The intent was that for a flight test, I’d preset the commanded roll and pitch values in the flight computer – they would be a “guesstimate” at first (a slight negative roll angle knowing the effect of the tail rotor force). The expectation was that while the helicopter was in the air, I could update the commanded roll and pitch values via the keyboard until I found a stable position – the Neural roll and pitch controllers would keep the helicopter at those commanded attitude values. The laptop flight software would save those values for the next flight.
I’d already spent a lot of time on wiring up the system, designing the Neural controllers, “flight testing” on the test stand, etc. but the final success of stabilized flight in my backyard seemed to elude me. My Northrop boss had been great in giving me the time off (leave-without-pay – as I’d already burned up all my vacation time – but held my job for me) – but he kept asking “when are you coming back?!?”.
So it was on a Friday that I called him – “I need two more weeks – after that, no matter what, I promise that I’ll be back”. That was, of course, two more weeks without pay. But Northrop management had bent over backwards to accommodate my insane endeavor.
While I told him that it would just be two weeks, in my mind I assumed that at this rate, it would be at least 3, maybe 4 more months until the goal was achieved – and of course I’d be working like a mad scientist in the evenings and during the weekends (and I couldn’t neglect my family – they needed my time as well). So it was all quite a bit depressing.
Nevertheless I worked all day Saturday (14 hours), took Sunday off (sometimes you have to step away and get new perspective), and worked another long day on Monday. On Tuesday morning … the Neural control system was successfully stabilizing the helicopter in flight in my back yard – as shown in the video below. So instead of being months away, I was only 3 days away from the successful test flight. It was on Monday that I’d found what I thought was the problem – and proved the solution Tuesday morning in my backyard.
The Lesson – We Never Know How Close We Are to Success
The lesson here was that on that Friday, I believed that I was still months away from getting the system to successfully hover in free-flight (not on a test stand) – yet in reality I was only 3 days short of the objective. It was a reminder to me that we can never give up – because we never know how close we are to reaching our objective. Many people quit too soon – never knowing that they were just hours or days away from achieving their success.
The solution that locked in the successful first flight is discussed at the end of this article.
After performing a few more flight tests in my backyard with extended training landing gear – and being reasonably confident that the Neural control system was very stable, I asked an RC pilot friend to help me test the system in an area out in the country. The objective was to test the helicopter Neural control system at 10-30 feet altitude with no safety gear – and simply let it hover for extended periods of time (10 to 20 minutes). These tests would solidify my confidence level regarding the performance and stability of the Neural control system.
The experienced RC pilot would be on hand to take control of the helicopter, via RC transmitter, if an anomaly occurred. There was a safety switch at the end of the tether on the ground (the safety switch was wired to the helicopter via the tether) that gave the RC pilot full control over the helicopter for emergencies or only partial control during the testing of the Neural control system. In this mode the pilot just controlled the throttle / collective and tail rotor, so as to be able to raise and lower the helicopter in altitude while the Neural control system stabilized the helicopter about the roll and pitch axes.
The following videos are of the subsequent flight tests that we performed out in the country – the purpose was to continue to test the stability and performance of the Neural control system. Each video starts and ends with the Neural control system actively stabilizing the roll and pitch attitude.
Flight Test #1
Flight Test #2
Flight Test #3
The rest of this article goes into detail on the effort that was involved to make this happen – Neural controller design, avionics integration, problem resolution, etc.
Neural Network Control System
The original idea had been to demonstrate that a closed-loop Neural Network attitude control system could easily stabilize a helicopter (ergo a real-time flight control system) about the roll and pitch axes. It’s not an easy problem – try to hover an RC helicopter if you’ve never done it. Typically a “newbie” will tend to overcompensate on the joysticks and cannot maintain stability about the roll and pitch axes if his or her life depended on it. The best way to learn is on a computer simulator before trying the real thing – then the many crashes one will experience, during the learning process, aren’t a problem (better than destroying an actual RC helicopter).
Thus I had to break it down into a classical control problem – plant, feedback error, compensation signal, etc. And that’s before, of course, bringing hardware into the equation.
Control System Diagram
In the classic control diagram below, the system being controlled is the roll axis attitude error and roll rate of the helicopter (and this example applies to the pitch axis mode as well). The objective of the control system is to be able to quickly, and in a stable manner, zero out the roll attitude error and roll attitude rate.
The “plant” is the helicopter itself – specifically the helicopter dynamics. Starting on the left in the diagram below, a fixed commanded roll attitude value (could be positive or negative) is added to the actual roll attitude value to produce a “roll error” (the difference is the error). This “roll error”, along with the roll attitude rate, is input and propagated through a controller, which will attempt zero out both the roll error and the roll rate. In this case, the controller is a Neural Network, which then outputs a correction signal (compensation) – the servo actuator rate command, specifically a delta value. This is added to the current servo command value and is then sent to the actuator to update the servo position.
This action feeds into the helicopter dynamics as the main rotor dynamics are affected by the updated servo roll actuator motion (the main rotor disk rolls to the right or left). The resulting change in motion of the helicopter is measured by the attitude sensor. The sensor feeds this information back to the flight software – and the cycle is repeated at a rate of 50 milliseconds (20 Hz).
Neural Network / Hardware Diagram
A general schematic of the entire system, which shows details of the Neural controllers and the hardware, is shown below. A Crossbow (the company was acquired by Moog, Inc. in 2011) Attitude and Heading Reference System (AHRS) was mounted on the front of the helicopter – it provided stabilized roll and pitch data to the control system. Note that I used an RC tail rotor stability device to keep the tail steady – the main initial focus of the project was on the helicopter roll and pitch axes (one problem at a time). The measured roll and pitch attitude outputs were provided, at a rate of 20 Hz to two Neural Network modules – one performed roll control and the other performed pitch control. Each Neural Network then output the required correct servo motor step value given the attitude error and rate values.
It’s important to note that both Neural Controllers were identical – that is the Neural Network developed from roll test data was also used to control the pitch axis. Thus the Neural roll controller – that learned from the roll dynamics only, also easily handled the very different pitch axis dynamics. So while there were two Neural controllers – one for roll and one for pitch – they were identical.
In the above diagram, the Crossbow AHRS is colored gold – this was a later model. The image was taken from their website (many years ago) for illustration purposes. The actual AHRS used in this effort was colored black as you’ll see in the hardware images further down in the article.
Flight Test Schematic
The field flight test schematic is shown below. The flight software – including the Neural Network controllers, was coded in C (using a Borland C compiler), running in Dos 6.22 on an HP laptop. The 90-foot tether, which connected the laptop with the helicopter, contained two RS-232 cable sets – one received the data from the AHRS (on-board the helicopter), and the other sent the updated servo commands to the helicopter.
In the laptop, an input text file contained the preset commanded roll and pitch attitude values for the Neural control system which would attempt to maintain the helicopter at these preset commanded attitudes. While the helicopter was in the air, if it started to drift to the left for example, I would incrementally increase the commanded roll, via arrow keys on the keyboard, until the helicopter stopped and maintained a reasonable hover (so that it didn’t drift all over the place). The flight computer would save the updated values for future flights. So typically after one flight, I didn’t have to update the commanded roll and pitch values as the natural hover points had already been established.
Development Laboratory – My Home Garage and Office
All of the development was performed in my home – specifically in my home office, my garage, and my backyard. All of the costs came out of my personal funds as well – the RC helicopter, the avionics equipment, laptops, tooling, etc. The most expensive single item was the Crossbow AHRS at just over $4,000 (remember that this was in 2001). Try convincing your spouse of the value of purchasing a small black box, for $4,000, that appears to do nothing and is not useful inside the house!! Nowadays these kinds of systems can be purchased for just a few hundred dollars.
Flight Test Stand
A flight test stand – upon which the helicopter would be mounted, was used in the development of the Neural controllers and was also used for preliminary testing. The image below shows the test stand with some explanations of the parts.
The image below shows the helicopter during a particular test (airframe test only) on the test stand. The test stand served several purposes, included vibration testing of the AHRS, and for generating roll profile data for creating the Neural Networks that would be used as the roll and pitch controllers.
Flight Computer – My HP Laptop
There were two laptops dedicated to the effort – one made by HP and the other made by Compaq (this was just before the time that the two companies merged). The HP laptop was used as the flight computer for controlling the helicopter (laptop on the ground communicating with the helicopter in the air via a 90 foot tether) while the Compaq laptop was used for bench testing avionics and other hardware components.
In the image below, the Compaq laptop is shown performing a test of the Crossbow AHRS unit.
Avionics Integration – Phase-1
The image below shows the initial avionics set-up – yes it’s very primitive but when you’re doing something like this on your own – with your own funds, on your own time, and just need a prototype – it’s sufficient. And while I tried to be careful, I did make some mistakes – one time I miss-wired the power and ground leads and burned up a Pontech servo controller board – that was a bad day.
The image below shows the basic layout.
The image below shows a different perspective with the Crossbow AHRS unit attached. However, after doing a lot of testing with the helicopter on the test stand, I realized that the AHRS was going to need to be isolated from the airframe vibration. Needless to say, it can be unnerving having a helicopter’s main rotor disk spinning at around 1,700 rpm in your garage in close quarters (yes a 5-6 foot diameter main rotor disk spinning that fast can take your head off in a split second).
Avionics Integration – Phase-2
In this phase, because of balancing issues, I decided to attempt to make the avionics integration more compact (move the weight closer to the main rotor shaft) – the revised layout is shown below. In addition, the AHRS was mounted inside a metal box with foam pads to provide the previously discussed required vibration isolation. Yes it looks like a mess but everything was tied down pretty solidly – I just needed it to work for field testing.
Helicopter Flight Test Configuration
Since I wasn’t an experienced RC helicopter pilot, for backyard flight testing I used “trainer landing gear”, so that if I had to take control of the helicopter near the ground, I wouldn’t overcompensate on the joysticks and flip the helicopter over on its side (a catastrophic situation since the main rotor blades would hit the ground while turning at 1,700 rpm) – likely the extended landing gear skids would “catch” the helicopter and give me a chance to recover and upright the aircraft.
Neural Controller Design
What are Neural Networks?
The following is a definition from Wikipedia which I think is reasonable:
Artificial neural networks or connectionist systems are computing systems vaguely inspired by the biological neural networks that constitute animal brains. Such systems “learn” to perform tasks by considering examples, generally without being programmed with task-specific rules.
For this project, Feed-Forward Neural Networks were used – an example is shown below with two inputs, one middle (or sometimes called “hidden”) layer, and one output. The lines connecting the different nodes are called weights or gains. For example, the input to P1 (P = Processing element) is the sum of input X multiplied by w1 plus input Y multiplied by w2. Mathematically speaking, each processing element is a hyperbolic tangent function whose minimum and maximum values asymptotically approach -1 and +1 respectively.
The “learning” or “relationship mapping” is contained in the architecture of the processing elements and the interconnected weights / gains. The ability to learn nonlinear systems is derived from the nonlinear nature of the hyperbolic tangent processing elements.
Why Use Neural Networks for Control?
Well – what is the purpose Artificial Intelligence?
The purpose is to each a “system” how to not only perform a task or series of tasks but to also create effective new solutions for circumstances for which it was not trained.
The automotive robotic systems shown below are very complex – however, they can only perform very specific tasks for which they are programmed.
The idea with Artificial Intelligence is just that – the system has some kind of intelligence that enables it to make decisions for situations beyond its training regime.
I’d already had a lot of experience applying Feed-Forward Neural Networks to a variety of simulation and image-recognition applications with amazing success – thus this application was just the next step. Neural Networks could be used to handle more complex helicopter control problems such as handling a sling-load, maintaining stability in very strong gusting / turbulent winds, etc.
Training the Neural Network from Transient Response Example
The basic concept was to use a transient response (rapidly decaying sinusoidal wave) as the “behavior to learn or emulate” to build an example training data set for the Neural Network. In other words, teach the Neural Network that it should quickly dampen out attitude error and drive attitude rate to zero in the process. Examples of various types of transient responses are shown below.
The helicopter transient response data (relationship between the servo actuator command profile and the response of the main rotor disk which is measured by the AHRS unit) would be generated in the following manner:
1) Mount the helicopter (with avionics gear) on the test stand.
2) Bring the helicopter up to full power (just enough throttle / collective for takeoff).
3) Run a transient response curve through the roll servo (from the laptop computer) in order to get a decaying sinusoidal motion of the helicopter as shown in the image below. The AHRS unit would measure the roll profile which would be recorded by the laptop computer.
Once the data is recorded, an area of the data is sectioned off for training and the data is curve-fit. The illustration below explains the objectives in setting up the training data.
The raw transient response roll profile – generated while the helicopter was on the test stand, is shown below.
The next step was to select the training data window, as shown below.
The final step, before scaling for training, was to move the “target” roll value to meet the actual settled roll value and thus have a zero error condition at the end of the training set. As shown below, the error (in gray) was measured from a horizontal line just slightly above the zero attitude angle line.
For this effort I used the 1998 Matlab Neural Network toolbox (running on a 32-bit Windows operating system) – specifically the Levenberg-Marquardt (LM) training algorithm for Feed-Forward Neural Networks. As background, the Levenberg-Marquardt optimization algorithm is used industry-wide to solve all types of optimization problems quickly. It is known for its ability to produce robust, optimal solutions much faster than other similar algorithms. To put it mildly, it blows the doors off of all other training algorithms for Feed-Forward Neural Networks.
These days I still use Matlab’s LM training algorithm – it’s now part of the “Deep Learning” toolbox. But in addition I’ve started using the Pyrenn Levenberg-Marquardt training algorithm as well. Here is the link to their site with downloadable Python and Matlab code – https://pyrenn.readthedocs.io/en/latest/. In fact, my next blog article will discuss an effort I did recently to compare performance of the two LM training algorithms.
Performance Shaping Technique
A special technique, that allows the user to adjust performance in real time (that not many people know about), is called “Performance Shaping”. It gives the user the ability to “dial in” various degrees of performance depending on the desired or changing performance requirements. This feature adds a measure of additional adaptability for changing conditions (especially for those for which the Neural Network was not trained).
The Performance-Shaping (hereon referred to as PS) capability is integrated by adding two converging lines that provide an envelope around the transient response. This tells that Neural Network that the PS values drive the required performance. Thus when the Neural Network is in active operations (post-training of course), it can be commanded to increase (tighten) or decrease (loosen) performance by changing the PS input values.
An example of how the PS technique was implemented in the helicopter controllers is shown below. Under normal conditions in this phase of testing, I never needed to adjust the PS parameters – but the idea was that down the road, in situations like gusting winds, the PS capability might be useful.
I’d previously developed this technique on a simulation of the classic cart / inverted-pendulum simulation (used in academia to understand coupled, nonlinear control problems) – and it worked very well (amazing actually). An illustration of the classic cart / inverted-pendulum system is shown below.
In the simulation, the PS Neural Network could be commanded to quickly upright the pendulum, and then walk the cart back to the original position (while keep the pendulum straight up). Or it could be commanded to do the opposite – get the cart quickly back to the original position while stabilizing the pendulum in the upright position. So performance was either emphasized for the cart (minimize the displacement error quickly) or for the pendulum (upright and stabilize the pendulum quickly).
The plots below are from the cart / inverted pendulum simulation – the performance curves for varying degrees of PS commands are shown. When the PS values were adjusted to command a highly damped transient response for the pendulum, the Neural Network did just that and walked the cart back to the origin more slowly. When the PS values were adjusted to commanded a highly damped response for the cart, the Neural Network quickly moved the cart back to the origin while taking its time stabilizing the pendulum.
The reason I was stuck on not getting the Neural controller to work (per the story that starts at the stop) was that at the end of the transient response curves, the rate curve did not end at zero – instead there was a small amount of data that reached above zero. For some reason I didn’t notice this and it was on that Monday that I noticed that the rate curve didn’t end with a zero value but instead ended with some positive value (was not very obvious until I looked more closely at the data). So what this did was teach the Neural Network that the target attitude rate didn’t need to be zero but could be some number about zero. And thus the controller was very sloppy and loose (I’d noticed this on the test stand but assumed it was some interaction with the test stand).
So I fixed the problem by manually making the rate end at zero (yes I altered the data near the end to fit the desired condition). Then I built several new Neural Networks with the slightly updated training set and selected the best one based on test stand performance. The following morning is what you saw in the first video – the correction / update worked beautifully.