The Grail Quest
To enable a future where everyone has access to instant delivery of exactly what they need when they need it, there is one fundamental technical challenge that has never been solved before.
Countless companies and research groups have embarked on this quest over the last 20 years. That quest is to find a solution to efficiently and reliably enable drones to detect and avoid other aircraft in the air.
Why Can’t We Do What We Do Today Everywhere?
Our national scale drone operations today are in countries who have modernized their airspace to require all air traffic to carry transponders and have scaled their Air Traffic Control operations to efficiently oversee drone operations in their national airspace. But many countries have not made these transitions, and it is left to the drone itself to provide the separation from other air traffic.
So Why Don’t Drones Just Listen to Transponder Signals from Other Aircraft?
Most people are surprised to learn that some aircraft flying in the United States do not carry a transponder. A transponder is a radio that transmits an aircraft’s position to surrounding aircraft. You have probably seen a movie where the autopilot alarm starts and a robotic voice tells the pilot “pull up, pull up” because they are coming close to another aircraft. That is based on the transponder of the other aircraft broadcasting a digital description of where it is on a radio channel. While transponders are required by regulators on many, including commercial, aircraft that fly at high altitudes, they are not required on many aircraft in most of the low altitude airspaces where drones fly here in the United States.
The Line of Sight Barrier
In the United States today, drone delivery operations “solve this” by flying with people on the ground within about one mile of the drone at all times who are looking for other air traffic in the sky. This is a fundamental barrier to scaled drone operations for three reasons: It is phenomenally expensive; people cannot see through clouds; and people looking up at the sky are actually not great at separating drones from other aircraft at long ranges looking in every direction — it is just really hard to do all day, every day.
Sensing and Avoiding
For the approximately two thirds of aircraft that carry transponders, our detect and avoid system receives their radio signal and uses that to stay well clear of those aircraft. But for the other third of aircraft, we need a way to safely and reliably sense and avoid them in order to fly the complex, scaled operations that expand communities’ access to the products they need. For decades companies (including Zipline) have tested all the obvious sensing options and run up against their disqualifying flaws. Let’s walk through a few of those options:
Radar has two huge problems. If you only need to see straight ahead, what is directly in front of the drone, there are radars you can fit on small drones. These radars have a huge problem seeing aircraft below the horizon, when that aircraft is between the radar and the ground. This is a failure mode that plagues forward looking radar for automotive driver assist systems. But the detect and avoid solution does not just need to look straight ahead.
360 Degree Sensing
If you are flying in a two seater aircraft, and you come up on a helicopter it is your job to see the helicopter and avoid it. Currently, most regulators expect drones to see aircraft coming from any direction and get out of the way. This requires a sensing solution that can see 360 degrees. To get long-range, 360 degree coverage from a radar you would need substantially more radar equipment, in terms of weight, than the entire drone weighs. And that is a problem for radar on top of the problem stated above.
Many companies have tried to solve this problem with cameras. And while wrapping a drone in cameras to get 360 degree coverage is more practical than radar, cameras suffer from two disqualifying flaws. They can’t see through clouds. We fly in and out of clouds every day — medical emergencies don’t wait for the weather to clear. The second flaw is a bit geeky. When you are trying to detect an aircraft that is really far away, in a camera it looks like a dot that may only be a handful of pixels wide. If you are on a collision course with another aircraft, there is a geometric problem that happens, where that aircraft on a collision course is the one dot in your camera image that is not moving, it is just getting larger, very slowly. Other aircraft with which you are not on a collision course will be moving dots in the camera image. This means that the one dot that is not moving in your camera image, is either a smudge on the camera or the one aircraft with which you are on a collision course.
Many other options have been explored and ruled out like infrared cameras that are too low resolution to see aircraft at sufficiently long range, or LIDAR that measure using tiny sparse laser dots where the spaces between the dots are really large — meaning the laser dot is likely to miss an aircraft all together.
We Went Back to the Drawing Board
As is baked into the DNA of Zipline, we went back to first principles and went through every possible sensor developed. We kept coming back to a crazy idea. Back in World War I, before the invention of radar, people would listen through giant “ear trumpets” that enabled them to hear aircraft well over the horizon, because, as we all know, traditional crewed aircraft are pretty noisy.
World War I Devices used to listen for aircraft coming over the horizon. Source
The idea of using microphones to listen for surrounding aircraft has a few key advantages. Microphones are lightweight enough that we could actually fit as many as we needed on a drone. And a microphone array could easily hear aircraft coming from any direction getting us that critical 360 degree coverage. But all aircraft are not loud, like gliders and hot air balloons. Could this still work? We dug into the data on the prevalence of different types of aircraft we expected to encounter and it turns out that about 98.5% of aircraft that don’t carry a transponder make plenty of noise.
And if we could detect anywhere near 98.5% of aircraft we would be much better than the status quo of pilots’ ability to see and avoid aircraft by looking out their windows. This convinced us that we had to explore this idea.
At Zipline, when we are going to explore a crazy solution we start by figuring out the hardest technical challenges that are most likely to kill the solution and we focus on solving just those challenges, or learning why, at a physics level, they are not solvable. We identified three such technical challenges to this solution:
- Over the sound of our propellers, which are so close to our microphones, could we hear the sound of far away, relatively faint aircraft?
- Could we hear far away aircraft over the aero acoustic noise caused by air flowing over our aircraft? Aero acoustic noise is the sound that gets really loud if you stick your head out of a car window but gets quiet when you bring your head back inside (don’t try this at home :)).
- Could we make microphones that would not get drowned out when getting pelted by moisture droplets ranging from dense but tiny droplets that make up clouds to large raindrops?
After a year of building prototypes, running tests and doing lots of analysis, we solved all three of these hard problems and sensing aircraft with microphones went from a crazy idea to a brilliant solution.
The next two years of development went into turning this de-risked idea into an operations ready technology. This time was spent on software systems to detect aircraft, localize where those aircraft are, avoid those aircraft as well as details like automatically detecting degradation in our microphones and automatically testing the microphone array preflight. This time was also spent on hardware design to integrate the microphone array and compute solution into our drones as well as to extensively test for reliability and performance so that we can partner with regulators to introduce the system into operations.
Now these microphone arrays are flying at some of our locations in “passenger mode,” which means that microphones and software are running on, but not yet controlling the aircraft. Once fully approved by safety regulators, Zipline onboard, acoustic detect and avoid will enable a future where everyone has access to instant delivery of urgently needed items exactly when and where they need them.
How Does It Work?
For an in depth technical overview of how this system works, please find our International Conference on Robotics and Automation keynote