This post is a follow-on from the previous one on this subject – here.
I’ve never managed a board or chip development that didn’t benefit from test/field experience and subsequent revisions. Usually the revisions are to lower the cost and raise the performance, and a range of defect corrections and perhaps feature enhancements go along for the ride. Although I’m not flying yet, I’ve made a point of doing a lot of “ground testing” of the EFI power system, and to that end I have a complete duplicate system set up in my office/lab here, along with an engine/ECU emulator that behaves exactly like the real thing. This post describes a PCB spin I’ve done, and the addition of the one major piece of software that I previously skipped – logging.
Board Spin
I had a small punch list of changes to perform, and during the second half of May, with the aircraft in the Hangar and cold winter weather setting in, I spent some time (mainly on the really cold days) in my comfortable heated office doing a minor board revision, with the following changes:
- Manufacturability corrections – bringing components and footprints into line to avoid unnecessary redundancy or duplication.
- External filtering control – I noted that, with long dedicated wire runs from the batteries (located aft of the baggage compartment in the RV-10), IR drops during left coilpack events could have an observable impact on the captured waveforms for the injectors. As an option I could use some external large / high-reliability capacitors to smooth things out, but these need to be fused and they need inrush current limiting to keep things under control at turn-on. Some board level additions were made for this option.
- Selection of all trip modes available on the board, controllable via the stuff list. I’ve experimented quite a bit with the various protection options of the TPS devices. I’ve decided that, for the injectors, I don’t want to use anything other than automatic over-current protection and retry, because I don’t want anything on the board that has a failure mode, regardless of how remote, which can disable an injector. For the coilpacks and fuelpumps, however, I’ll leave the ability to set the over-current protection to manual, because there are two completely independent sets of plugs and fuelpumps. As a result, (i) an extremely unlikely board level failure mode which can disable one of these supplies would not be catastrophic, and (ii) in the event of a real fault which causes over-current on one of these channels, in the case of a bad short circuit the retry trip current can be a 20 Amp spike every few msec, and although the power system is perfectly happy doing this it looks very ugly and probably over time takes a toll on the associated battery/alternator system. It might also lead to RF interference. As such, being able to disable the individual power circuit in-flight probably has some merit.
Once the new PCB’s arrived, I hand assembled one across a quite a few sessions which probably totaled about 16 hours of microscope time. The entire board came up fine on first power-on, and the new features were all good. After testing I’ve decided to do one last board change, which incorporates the one remaining and embarrassingly obvious feature enhancement:
- Use a separate wire run from the common left EFI supply fuse for the injectors, so that the existing fuse which is right next to the left/primary battery has two separate wires running to behind the panel for the EFI power system. The existing wire is for the left coilpack and left fuelpump power. The new wire (same gauge) will be for the left injector power. This largely prevents the coilpack pulses and fuelpump current from impacting the measured fuel injector pulse data, and is a much better method of isolating these observations from each other than the use of external filter capacitors.
Here’s a picture of the Rev 1.1 board on the lab bench, a bit messy due to a bunch of test points and the debug interface. I’m no longer stuffing the RS-232 interface, the USB-C link is fast and reliable and that’s how the aircraft is wired.
Logging
The logging requirement specification is simple:
- I want to log every event that occurs with the power system loads, at the full resolution available from the underlying sampling system. That means every injector pulse, every coilpack pulse, the fuelpump ON/OFF state at the time of any change, and a regular sample of the fuelpump dynamic power characteristics (read: commutator noise). For the LCD display, this data is scaled and down-sampled in order to fit onto the display. For logging, I want to maintain the full 12 bit ADC accuracy of the measurements, and the full sampling resolution, which in the current hardware is one (4x over-sampled) measurement for each of the ten power channels (6 ignition + 2 coilpack + 2 fuelpump) every 52 micro-seconds.
- Logging data is to be dumped to a removable flash drive during flight (plugged in to the CPU under the pilot’s seat), so that this flash drive can be removed after flight (by sliding the seat back) and downloaded, or alternatively dumped across a wireless network.
- The presence of active logging is not to compromise the features, quality or refresh rate of the LCD display.
- Log data must include precise start/end timestamps, so that the entire log can be matched up with other engine log systems, such as data available from the SDSEFI ECU, and/or the EFIS related engine monitors.
I can’t simply log every sample. To do so would require a data link with bandwidth of at least 3.9 MB/sec, and the system would accrue log data at the rate of around 14 GB per flight hour. The main focus of the existing firmware is to locate and sample pulses (injectors, coilpacks) and on/off changes (fuelpumps), so it seemed logical enough to extend this functionality to include real time logging.
Long story short – with the addition of a few thousand lines of firmware code, communication protocol additions, and host software additions, the logging system is now fully operational. Pulses are timestamped and a simple lossless data compression scheme is used to reduce the data requirement. Log data is stored locally on the power board and periodically downloaded by the host system, at up to 20 times per second. Apart from various test modes, the usual method in place is that logging will start any time either fuel pump goes on (from both initially off), and stop once both fuel pumps are off and the backlog of log data has been downloaded. Each logging start event leads to a time stamped directory on the flash drive which contains all of the log data. A timestamp file from the host system provides high resolution start/finish times, which can be used to correct any slight inaccuracy/tolerance in the power board’s clock. Compressed log data accrues on the flash drive at a rate of approximately 300 MB per flight hour. I bought a 256GB USB-C flash drive for about $60, which will fit log data for over 800 flight hours before requiring some sort of intervention, that’ll do.
I’ve tested this system extensively on the bench and it is now rock solid, with no noticeable impact on LCD display response or refresh rate.
Viewing log data
This is an entirely different topic, and a complex one. A two hour flight leads to a large amount of log data. Viewing this “big data” efficiently, being able to rapidly zoom in to a point of interest to examine it in more detail etc. is a common and not very well addressed real-world problem in the IT industry. I didn’t want to devote much time to this right now, so for the time being I settled on being able to extract anything up to a 5 minute slice of log data from a larger database, and view this in a zoom-able graphical form with some Python code. Here’s a zoomed-in example (injector loads are resistive in this sample (keeps the lab quiet), and no fuel pumps were active in this test):
Log data isn’t primarily for “viewing”. I think where this will end up is with a set of heuristics, some applied in real time as log data is collected, and some performed by the host CPU after engine shut down, i.e. after each flight. Any anomaly found will lead to some sort of notification message, which on the ground can lead to an actual human examination of the log data and if necessary a maintenance action. Numerous issues come to mind, for instance:
- An injector has started to go a bit sticky and occasionally not open – as determined by an (infrequent) lack of an inflection in the rising edge of the injector current.
- A fuelpump has started to show signs that a commutator segment is failing
- Ignition (coilpack) characteristics have changed, or intermittently change, indicating degradation of either the coilpack, drivers, or an ECU.
This sort of capability is a project for the future, after the aircraft is flying, where the necessary algorithms can be developed using actual flight logs. Where analysis in real time is possible, some of these heuristics can be migrated in to the real time logging system. The whole point is to provide early detection of impending failures, and to fix the associated problem before it becomes a hard failure, and before leading to a noticeable change in engine performance. For example, a fuel injector that previously worked every time now fails to open once every thousand or so cycles.
As usual with this activity, I’m logging zero hours, because it isn’t really a normal part of an RV-10 build.