AVs Will Need Teleoperation: Here’s Why
Teleoperation for autonomous vehicles (AVs) is growing in importance. This column will explore the AV teleoperation segment and will give you information, perspectives and opinions on the following questions:
• What is teleoperation? What are the levels of teleoperation?
• Why is teleoperation needed for AVs?
• Where are AV teleoperation regulations emerging?
• What are the AV teleoperation use-cases?
• What are the teleoperation startups?
• Is teleoperation a make or buy choice?
• Will there be AV teleoperation standards?
• Any important organization with impact on teleoperation?
There is limited information available on AV teleoperation. The best information source was a teleoperation conference that was held by National Institute of Standards and Technology (NIST) in November 2020. Another good information source is the Teleoperation Consortium that was formed in December 2020. Both are described in more details below.
The next table has summary answers and comments to these questions with more discussion and information below the table.
What is teleoperation?
Teleoperation is defined as “operation of a system or machine at a distance” by Wikipedia. It can also be called remote operation or remote control. Teleoperation requires a communication link between the teleoperator and the remote machine — either a wireless link or a wired link. A wired link limits the range to nearby operation.
The history of teleoperation goes back to the 1870s with inventors working on remotely operated weapons. The first practical wire guided torpedo patent was reportedly issued in 1877. In 1898, Nikola Tesla demonstrated a remotely controlled boat with a patented wireless radio guidance system. We have come a long with a variety of teleoperated machines ranging from remote controlled toy cars and the rapid growth of teleoperated drones — from toys to delivery drones.
Levels of AV teleoperation
Teleoperation of AVs must be wireless and needs a very stable and reliable connection link. There seems to be agreement that there are three levels of AV teleoperation:
- Remote monitoring of AVs. Live monitoring of how an AV fleet is driving. It can also provide lots of useful information to improve the AV software quality and capabilities.
- Remote assistance to AVs. This level has a range of services with increasing remote assistance. The first case provides information and context to driving issue. The next case is transferring tailored driving data to the AV, which the AV completes. The highest case is short-time remote driving of the AV. Remote AV assistance sounds like what is needed for resolving edge cases.
- Remote control of AVs. This level usually includes remote driving for a substantial time. It includes real-time and direct control of the AV. This level seems like what is needed for hub-to-hub autonomous truck use-cases. In this case a remote driver controls the truck from the start hub to the highway, the AV software driver is the pilot on the highway, and the remote truck driver controls the last step from highway to destination hub.
There seems to be agreement that AV teleoperations is only needed for L4 and L5 autonomous driving.
Why is teleoperation needed?
Teleoperation is needed as a backup to driverless vehicles and is expected to be part of AV regulation in most countries. Teleoperation regulation is emerging and is discussed below.
Teleoperation is needed help to solve unknown driving situations — the so-called edge cases for AVs. This will happen less frequently as the AV software driver capabilities improves for each of the AV use-cases. The goal and strategy are to turn edge-cases into known driving events that the AV software driver can handle when encountered in the future.
The third reason is to gain early AV deployment capabilities with acceptable safety for some use-cases. Such use-cases are discussed below.
Teleoperation regulation status
California included teleoperation as part of its regulation for driverless vehicles in 2018. Multiple states have mandated teleoperation as part of their AV regulation including Arizona, Florida, Michigan, Ohio and Texas. Other states such as Georgia, Nebraska, Nevada, North Carolina and Tennessee have laws allowing teleoperation, but they do not mandate it specifically.
Several countries have added teleoperation to their AV regulation including Canada, Finland, Japan, Netherlands, Sweden and UK.
Shanghai and other Chinese cities have teleoperation as part of their AV regulation. Baidu Apollo includes teleoperation as part of its software driver and calls if 5G Remote Driving. I am quite sure that China will mandate teleoperation in its AV regulation.
Teleoperation is already used in multiple AV use-cases. Currently sidewalk AVs and industrial AVs are the most common teleoperation use-cases. Examples of sidewalk teleoperation is Kiwibot and Postmates.
The industrial teleoperation AV use-cases include forklifts, excavators (mining), combines (agriculture), yard trucks (ports and similar loading/unloading operations) and other material handling applications. Industrial teleoperation is a leading use-case because teleoperation has additional advantages in human safety and better driver utilization.
Autonomous trucks will be an important teleoperation use-case. Einride is one example where teleoperation is part of the AV systems. Hub-to-hub autonomous trucking is expected to see strong future uses.
Robotaxi operation without safety driver is expected to use teleoperation as part of the AV software driver. Zoox is an example and it has patents on teleoperation.
Another example, which is somewhat surprising to me, is shared electric scooters. The use-case makes a lot of sense. Teleoperation is used to move the electric scooters to charging stations and to return the scooter to home base after rental use. Both applications will increase operational efficiency and save costs.
I have included information on four teleoperation startup that are listed alphabetically. All of the teleoperation are recent startups—four years or younger.
Designated Driver was founded in 2018 and provide teleoperation of autonomous and non-autonomous vehicles. It offers three products: 1. In-vehicle hardware and software kit that can be integrated with different vehicles. 2. Driver Station hardware and software kit for the remote tele-operator. 3. Teleoperation-as-a-service for fleets including trained and certified remote drivers, which also includes in-vehicle teleoperation software.
The only publicly available information is a deployment with Texas A&M university to provide teleoperation for fixed route shuttles starting in August 2019. Designated Driver products are available via AutonomouStuff, a company that provides a large variety of AV products.
DriveU was founded in 2019 and provides a teleoperation platform that is deployed and used on public roads. The platform has both remote driving and remote assistance teleoperation capabilities. It is available as a software-only platform or with hardware and software. The platform offers high speed transmission of 4k video, low latency, audio streams, high-speed data, and control channels. The DriveU platform provides system integration via SDK and APIs.
In January 2021, DriveU helped form the Andromeda Consortium for AV teleoperation. The consortium was formed by the Israel Innovation Authority and is composed of universities and leading Israeli companies. It is focused on legal and technological aspects to help speed up deployment of AV fleets through teleoperations. The consortium has a planned budget of $17 million. Over the next three years the consortium, will focus on advanced technologies for deployment of teleoperation of AV fleets, including robotaxis, delivery robots and autonomous trucking.
In April 2021, DriveU announced it is working with Intralink to provide AV teleoperation deployments in the Chinese AV market.
Ottopia was founded in 2018 and is headquartered in Israel. It has received VC investments of $12 million with Hyundai as an investors. Hyundai was the lead investor in the latest investment round in April 2021.
Ottopia offers both assisted teleoperation and remote driving. Ottopia’s strength is its teleoperation software platform that leverages AI to predicts each cellular carrier’s quality along the AV’s route. Ottopia also has extensive experience in video compression to lower the amount of video that needs to be transferred. Ottopia’s goal is to provide an automotive-grade teleoperation platform—including cybersecurity.
Ottopia’s product include an in-vehicle teleoperation module and a teleoperation center console. The in-vehicle module is a dedicated hardware unit that runs proprietary software and supports all teleoperation use-cases. The in-vehicle module has extensive cybersecurity that is tested against known attack vectors. The in-vehicle module includes the Advanced Teleoperator Assistance Systems, which has safety algorithms and the Advanced Teleoperation APIs for scalable control methods.
The teleoperation center console features off-the-shelf hardware with specialized software to run a teleoperation center. It includes shift management tools for a lean operation and extensive capabilities to handle video streams fused in real-time with vehicle and sensor data.
Ottopia’s partners include Bestmile, BMW, Deutsche Telekom, Denso, EasyMile, Gaussin, Innoviz, May Mobility and Via. With Hyundai investing in Ottopia in April it is likely that Hyundai may be a future partner.
Phantom Auto was one of the pioneers in AV teleoperation and was founded in 2017 and is headquartered in Silicon Valley and has an R&D center in Tel Aviv, Israel. It has received over $20 million in VC investments.
The Phantom Auto teleoperation platform is vehicle-agnostic with low latency communication software, an API for real-time driving, guidance and assistance, and remote operator training services.
Phantom was initially focused on opportunities in AV segments such as robotaxis. Phantom is the supplier of teleoperation for Postmates sidewalk AVs. As the deployment of these AV use-cases were delayed, Phantom shifted its attention adjacent segments that could use its technologies. Now Phantom has two key segments—material handling such as a variety of forklifts, and yard trucks in intermodal shipping areas.
Phantom Auto is working with customers in the U.S. and other regions with multiple vehicle types. Recently, Phantom has been most successful with industrial vehicles such as forklifts and signed a new customer, French logistics firm Geodis, in France in March 2021. Geodis and Phantom Auto have been working for over two years via a pilot programs in multiple French cities. Geodis is a major supplier of forklifts.
Phantom teleoperation can also be used on robotaxis, autonomous trucks, AV shuttles, goods delivery robots (sidewalk AVs) and industrial vehicles such as yard trucks.
Phantom has announced a partnership with multiple companies. 1. ITS ConGlobal (ITSC) is a large intermodal service provider in North America. ITSC operates rail and port terminals and provides freight infrastructure to major railroad companies, including Union Pacific, BNSF and Norfolk Southern. 2. Terberg needs Phantom’s remote operation to supplement all types of vehicles for yard automation. 3. FANUC’s robotic arms automatically help to connect and disconnect air lines from yard trucks to trailers. Phantom’s partnership with FANUC enables its customers to fully remove humans from the operation.
Teleoperation strategy: Make or buy?
Teleoperation strategy will be a mixture of make and buy. I think the major AV software platform companies will develop their own teleoperation software platform. There will also be good opportunities for teleoperation startups as many AV fleets and logistics companies will prefer to buy teleoperation software. Some AV software companies may also buy teleoperation software and make modification to fit their operational needs.
There are eight companies that have received California driverless licenses, which require teleoperation. The eight companies are listed alphabetically: AutoX, Baidu USA, Cruise, Nuro, Waymo, WeRide and Zoox. I have verified that two more AV companies are using teleoperation systems: EasyMile and Nissan. There are probably others.
Teleoperation is likely to have standards but what type of standards is not clear. The Teleoperation Forum, covered below, had a session on teleoperation standards with multiple speakers and lots of opinions. The next version of SAE J3016 (the SAE definition of AVs L0 through L5) will include definition of teleoperation, which is likely to be the starting point for teleoperation standards.
I think teleoperation standards will happen at several levels. The ley question is what levels will the standard cover? I think the best chance is for high level standards. I also think NIST will play an important role in getting teleoperation standards done in the United States.
A virtual teleoperation conference that was held by NIST in November 2020 has excellent information. There were 40 speakers—a who’s who of teleoperation—with video session covering over 40 hours available. The presentations are available at NIST Vehicle Teleoperation Forum | NIST.
An organization focusing on teleoperation was formed in December 2020 and is likely to play an influential role in teleoperation. The website is: Teleoperation Consortium. TC is a non-profit business organization located in Michigan. The TC mission is to provide communication, collaboration and consensus for the teleoperation industry. TC currently has over 30 members.
Teleoperation is on its way to play an important role in autonomous vehicles and their use-cases. I believe all AV regulations will require teleoperation or will allow/encourage teleoperations. Most of the AV software platform companies have their developed teleoperation software platform to meet emerging driverless AV regulation.
There are important teleoperation startups offering software platforms with some having started teleoperation deployments. The business model will be SaaS and cloud-centric operation that will have attractive financial performance when volume deployments start.
For the AV software platform companies, the business model is both a cost saving business model and the opportunity to start deployment earlier and safer than without teleoperation.