How John Deere created its autonomous tractor

Adding new brawn to the self-driving vehicle industry, John Deere unveiled a 40,000-pound autonomous tractor at this year’s Consumer Electronics Show that it says will be commercially available by the end of 2022.

The system uses six pairs of stereo cameras combined with GPS guidance to drive a Deere 8R tractor with a chisel plow and the capability to tow other equipment. A farmer can put the tractor to work with a swipe of a smartphone app and then walk away to spend time with family or attend to other business, using the app to monitor the tractor’s progress plowing a field or performing some other task — and to receive alerts of anomalies the software doesn’t know how to handle. While it’s working, the tractor can also gather data about the health of crops in the field, the health and moisture content of the soil and other metrics.

Jahmy J. Hindman, chief technology officer for Deere & Company, celebrated the announcement as a milestone for farm productivity. “Until recently, agriculture has always been about doing more with more — more horsepower, more inputs, more acres — but the new digital era is changing all that. The last decade has been about doing more with less and providing farmers with additional tools to make more informed decisions,” he said.

Self-driving cars have been a staple of CES for the past several years, and this year’s show featured exhibitors offering components such as LiDAR (light detection and ranging) sensors for autonomous driving and driver assistance systems. However, the Deere tractor doesn’t use LiDAR, and in general, the company’s technologists were not able to port hardware or software from the self-driving car world to their application. As a computing environment, a tractor is fundamentally different from an automobile in terms of vibration, temperature, dust, and other challenges. So the stereo cameras, for example, are Deere’s own design, he said.

Hindman said Deere is able to start with some off-the-shelf components, but invariably must customize them for its applications. For example, the tractor uses Nvidia Jetson Xavier GPUs but with a custom assembly for passive cooling in a dusty environment where conventional computer fans wouldn’t be practical.

“As for the software side of things, that’s all us and all-new,” he said.

To put itself in a position to explore applications of AI to agriculture, Deere spent $305 million in 2017 to acquire Blue River Technology.

Blue River provided the technology behind Deere’s See and Spray technology for the application of herbicides, which it says can reduce the amount of chemicals sprayed on a field by approximately 80%, saving farmers money and promoting more sustainable agriculture. Also based on computer vision with stereo cameras, See and Spray distinguishes between plants and weeds as a sprayer passes over them to ensure chemicals are sprayed only on the weeds.

While Deere’s application is custom, it certainly takes advantage of more general advances in the state of the art for AI, such as common architectures for deep neural networks, said Willy Pell, the vice president of autonomy and new ventures with Blue River. The Deere subsidiary is about 30 people, he said.

For the autonomous tractor project, Deere’s technologists loaded prototype tractors with all sorts of sensors and ran them over fields, logging data to see which ones provided the most useful information, Pell said. “We landed on stereo cameras, which was unintuitive because we thought we were going to be doing LiDAR,” he said. Lidar turns out to be excellent for detecting other vehicles on the road ahead, along with their distance, direction, and velocity, but less relevant for a tractor navigating through acres of open farmland.

In contrast, stereo vision cameras produce good depth perception, and Deere’s software is able to break the image into multiple views — the raw image, a depth map, and per-pixel classification of each part of the image that distinguishes between ground and sky and between a crop and a foreign object that may have fallen onto the field.

The tractor is programmed on how to handle common occurrences, like the need to navigate around animals that might be crossing a field, but it also includes an anomaly detection system to help it cope with the unexpected. For example, if it detects an object in its path not accounted for in its training data, it simply stops and signals the farmer to decide what should be done.

Deere emphasized the care they have taken to make sure the machine can operate autonomously but also safely.

This is one example of where a self-driving tractor has an advantage over a self-driving car. “When we encounter an anomalous object, we stop — we don’t have to worry about getting rear-ended by another driver,” said Deanna Kovar, vice president of production and precision agriculture systems at Deere.