SYSTEMS, DEVICES, AND METHODS FOR ROBOTIC REMOTE SENSING

FOR PRECISION AGRICULTURE PRIORITY CLAIM

The present application claims the benefit of U.S. Provisional Patent Application Serial No. 62/108,509, filed January 27, 2015, the disclosure of which is incorporated herein by reference in its entirety.

GOVERNMENT INTEREST

This invention was made with government support under N00014-07- 1 -0829 awarded by the Office of Naval Research and under 1 1 13830 awarded by the National Science Foundation. The government has certain rights in the invention.

TECHNICAL FIELD

This subject matter disclosed herein relates to systems and methods for data-driven remote sensing for precision agriculture. Specifically, the subject matter disclosed herein relates to the development of imaging systems and deployment modalities for close-range sensing of critical properties of specialty crops, such as apples, oranges, strawberries, peaches, and pecans.

BACKGROUND

Persistent and timely monitoring of agricultural farmlands have shown to be increasingly valuable to crop health and resource management. Remote sensing satellites and airborne sensing with winged aircrafts have allowed scientists to map large farmlands and forests through acquisition of multi-spectral imagery and 3-D structural data. However, data from these platforms lack the spatio-temporal resolution necessary for precision agriculture. For example, a typical remote sensing satellite image may have a pixel resolution of hundreds of meters, and airborne sensing may provide resolution of a few meters. It is desirable, however, to obtain data for monitoring orchard or vineyard health at a centimeter scale - a resolution at which stems, leaves, and fruits can be observed.

As a result, farm management tasks such as yield estimation and disease monitoring are primarily carried out through visual inspection by human scouts. Recent development in this area has resulted in imaging systems and data analysis methodologies to help automate some of these tasks (See, e.g., U.S. Patent Application Pub. No. 2013/0325346). For example, unmanned ground vehicles (UGVs) have been the first step towards automating the close-range monitoring of high-value crops. They can carry a variety of bulky sensors such as LiDAR for volumetric mapping, and ground penetrating radar (GPR) and electrical conductance sensors for precise soil mapping. Due to the mobility constraints of unstructured farms, however, it is infeasible to use UGVs for rapid and persistent monitoring. Additionally, ground vehicles are intrusive. Aerial platforms and hand-held sensors can alleviate some of the problems with using UGVs, but the available platforms for such systems are bulky and expensive, which can be prohibitive for large-scale deployments in farms. Furthermore, the spatio- temporal resolution of such systems are considered inadequate as discussed above.

Accordingly, it would be desirable to develop a portable, low-cost, compact, and lightweight imaging system along with agile deployment methodologies to help growers observe farms efficiently. As a part of this system, it can be advantageous to have a powerful data analysis and visualization framework to help growers interpret the acquired data.

SUMMARY

Disclosed herein is a methodology for data-driven precision agriculture through close-range remote sensing with a versatile imaging system, which may be deployed onboard low-flying unmanned aerial vehicles (UAVs), mounted to ground vehicles (e.g., unmanned ground vehicles (UGVs)), and/or carried by human scouts. Additionally, the present technology stack may include methods for extracting actionable intelligence from the rich datasets acquired by the imaging system, as well as visualization techniques for efficient analysis of the derived data products. The systems and methods discussed herein may include one or more of these four components: an imaging system, a deployment methodology, data analysis algorithms, and/or a visualization framework that, when used together, may help specialty crop growers to save resources (e.g., less fertilizers, water, and pesticides may be needed because of better stress and disease monitoring), optimize crop yield, and reduce costs (e.g., by allowing for a better allocation of labor due to the efficient estimation of crop yield and from the lower use of resources).

Although some of the aspects of the subject matter disclosed herein have been stated hereinabove, and which are achieved in whole or in part by the presently disclosed subject matter, other aspects will become evident as the description proceeds when taken in connection with the accompanying drawings as best described hereinbelow.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present subject matter will be more readily understood from the following detailed description which should be read in conjunction with the accompanying drawings that are given merely by way of explanatory and non-limiting example, and in which:

Figures 1 a and 1 b are top and side views, respectively, of a lightweight, low-cost, portable, compact, and self-contained multi-spectral 3-D imaging sensor suite designed for precision agriculture according to an embodiment of the presently disclosed subject matter;

Figure 2 is a perspective view of a UAV for precision agriculture according to an embodiment of the presently disclosed subject matter;

Figure 3a is a perspective view of a UAV in-flight at a vineyard according to an embodiment of the presently disclosed subject matter;

Figure 3b is a real-time 3-D map of the rows of grape trees imaged by the UAV shown in Figure 2;

Figures 4a through 4c are bottom, side, and wide-angle views, respectively, of a multi-rotor UAV with the sensor suite facing downwards according to an embodiment of the presently disclosed subject matter; Figure 4d is a front view of a sensor suite harnessed on a human scout according to an embodiment of the presently disclosed subject matter;

Figure 4e illustrates a harnessed sensor suite according to an embodiment of the presently disclosed subject matter being used to scan a row of dwarf apple trees at an apple orchard in Biglerville, Pennsylvania;

Figure 5 is a flow chart illustrating a data processing pipeline of the system according to an embodiment of the presently disclosed subject matter;

Figure 6a is a 3-D reconstruction of a row of grape trees spanning about 70 meters at a vineyard according to an embodiment of the presently disclosed subject matter;

Figure 6b is a map showing canopy characteristics and scene features from the reconstruction shown in Figure 6a;

Figure 7a is a multi-spectral 3-D reconstruction of a row of dwarf apple trees just using laser data according to an embodiment of the presently disclosed subject matter;

Figure 7b is a multi-spectral 3-D reconstruction of a row of dwarf apple trees with thermal data overlaid on a 3-D point cloud according to an embodiment of the presently disclosed subject matter;

Figures 8a through 8d are 3-D point clouds of canopies that may be used to determine tree height, canopy volume, and leaf area according to an embodiment of the presently disclosed subject matter;

Figure 9 is a graph illustrating a Correlation between LiDAR area index and measured true leaf area for the data points corresponding to the emulated canopy stages of two trees according to an embodiment of the presently disclosed subject matter; and

Figures 10a and 10b illustrate data from a thermal camera along with data from visible-range cameras according to an embodiment of the presently disclosed subject matter.

DETAILED DESCRIPTION

Systems, devices, and methods for robotic remote sensing for precision agriculture disclosed herein address the limitations of current farm monitoring practices through the development of a technology stack for high- resolution multi-spectral 3-D mapping of specialty crops. The present subject matter exploits a versatile sensor suite capable of being deployed in multiple modalities (e.g., mounted onboard UAVs or UGVs, and/or carried by human scouts). The target applications include but are not limited to yield estimation and disease monitoring for apples, oranges, strawberries, peaches, and pecans.

In this regard, the present subject matter provides systems, devices, and methods for data-driven precision agriculture through close-range remote sensing with a versatile imaging system. In one aspect, the present subject matter provides a compact multi-spectral 3-D imaging system. As discussed above, in recent years, there has been a growing interest in the use of imaging sensors for monitoring plant physiology and morphology. Accordingly, referring to one embodiment illustrated in Figures 1 a and 1 b, a sensor system, generally designated 100, may comprise an array of science sensors, navigation sensors, an onboard computer, a wireless communication link, and/or batteries. In this way, sensor system 100 according to the present subject matter may combine different sensing modalities on a self-contained, lightweight, and compact platform that can be implemented using any of a variety of deployment modalities. In particular, for example, the sensors onboard may be selected to monitor a range of plant physiological and morphological properties such as canopy volume, leaf area, water stress, and crop yield (e.g., fruit count, fruit size).

Specifically, for example, in some embodiments, sensor system 100 may include one or more laser range (e.g., LiDAR) scanners 101 , which may be configured to extract morphological properties (e.g., canopy volume and leaf area) of the agricultural subject. In one particular configuration, for example, LiDAR scanner 101 may be a Hokuyo UST-20LX laser scanner, which may provide high-resolution laser scans for monitoring plant morphology.

Furthermore, in some embodiments, sensor system 100 may comprise one or more thermal cameras 102 and/or one or more multi- spectral cameras 103 configured to provide imaging in the red and near- infrared bands. Imagery from such bands may be used to monitor plant vigor, which may in turn be used in guiding pruning management and fertilization. In addition, thermal cameras 102 and multi-spectral cameras 103 may be used together to estimate a range of plant properties related to photosynthetic efficiency and water stress. In one particular configuration, for example, thermal camera 102 may be a FLIR A35 thermal imaging camera, which may be used to collect temperature readings (e.g., at a resolution of 320x256). Multi-spectral cameras 103 may include two monochrome Matrix Vision BIueFox cameras equipped with narrow-pass filters (e.g., at about 670nm and 800nm), which may together provide the necessary bands for calculation of the Normalized Difference Vegetation Index (NDVI) - an indicator of plant vigor. In addition, sensor system 100 may further include an RGB camera 104 (e.g., an RGB BIueFox camera) to acquire true-color data. Data from the wide range of spectral bands may be used for fruit- counting and monitoring crop stress and disease.

In addition, in some embodiments, sensor system 100 may include one or more of stereo camera rig 105 for visual odometry and reconstruction and/or a spectrometer 106. One or more navigational sensors 107 (e.g., global positioning system (GPS) sensor or other navigational sensors) may be included in sensor system 100 to determine the location in geographic space. In some embodiments, for example, navigational sensors 107 may include a Microstrain 3DM- GX4-25 IMU, a μΒΙοχ precise point positioning (PPP) GPS sensor, and/or two single-channel BIueFox cameras for stereo visual odometry.

Further, in some embodiments, sensor system 100 may include an inertial measurement unit (IMU) 108. As used herein, the term inertial measurement unit (IMU) should be understood to include a sensor that measures linear accelerations and angular velocities in three dimensions. In addition, barometric pressure sensors and magnetic field sensors may further be included in IMU 108, although these may alternatively be provided as independent sensors (i.e. , a barometer and/or magnetometer).

An onboard computer 109 (e.g., Intel core i5 computer) may be used to log data from all the sensors and to facilitate communication to a base- station or the deployment vehicle (e.g., through a wi-fi link). Power may be delivered by one or more batteries (e.g., two 2700 mAh lithium polymer batteries).

Regardless of the particular components carried or their configuration, sensor system 100 may be configured to be relatively light (e.g., having a total weight of 1 .6 kg or less) and compact (e.g., having dimensions of about 40 cm x 13 cm x 13 cm). It may further include a strong but lightweight frame 110 (e.g., a carbon fiber frame) that supports a base plate 111 (e.g., a polycarbonate plate) on which all sensors are mounted. This sensor arrangement may provide reliable operation in different deployment modes. Specifically, for example, an endurance of about an hour has been observed during the deployments.

Since the disclosed array of sensors may provide a platform for performing detailed and highly accurate environmental reconstruction, in some embodiments, it may be desirable that both the state estimation and scientific sensors be properly calibrated. For example, prior to each deployment, a complete system calibration may be performed, wherein the stereo camera rig 105 and the sensing cameras (e.g., thermal camera 102, multi-spectral camera 103, and/or RGB camera 104) are calibrated relative to IMU 108. This process may include both camera-system calibration (e.g., stereo, color, and multispectral) and spatial/temporal calibration of the respective cameras and IMU 108.

In addition, the intrinsic and extrinsic parameters of thermal camera 102 may be calibrated relative to stereo camera rig 105. This procedure can be complicated by the fact that thermal camera 102 cannot observe visible wavelengths, and produces only a low-resolution image (e.g., 320x256). Thus a standard chessboard pattern may not be available to be used to calibrate thermal camera 102 directly. In order to address this problem, an ordinary circle grid pattern printed on paper may be illuminated by a hot lamp, producing a pattern which is discernible in both the long-wave IR and optical regions of the spectrum. This approach allows for calibration of thermal camera 102 itself as well as with other cameras without introducing any complicated calibration device. Regardless of the particular configuration of sensor system 100, the present subject matter may be deployed as a distributed plurality of sensor arrays. For example, the system may be mounted onboard one or more UAVs, generally designated 200, for rapid sensing in unstructured farmlands. As used herein, the term unmanned aerial vehicle (UAV) is understood to include an aerial platform that may be controlled remotely by a human or with onboard computers, and which may be capable of carrying payloads. UAVs 200 are well-suited for precision agriculture due to their small size, superior mobility, and hover capability, which allows them to perform both high-altitude remote sensing and close inspection (e.g., at distances less than 2m) of problematic areas. In this regard, it has been successfully demonstrated that UAVs 200 equipped with multiple sensors are able to navigate autonomously through complex indoor and outdoor environments. Because of this combination of flight capability and maneuverability, such UAVs 200 may be advantageously applied to precision agriculture to produce both aerial views and side views (e.g., between rows of crops rather than just above them) of the subject agricultural area. This ability to collect data from multiple perspectives allows for complex modeling of the farmlands with high resolution and without large, intrusive, expensive, and/or overly complex data collection systems.

Accordingly, multiple UAVs 200 may be deployed with sensor system 100 onboard to acquire richer data. Specifically, for example, multiple UAVs 200 may be used to collect data from different parts of the farm, or from different angles, and the fused data may be used to generate 3-D maps with larger spatial coverage. Exemplary UAVs 200 used with the present subject matter may be configured to be lightweight (e.g., about 10 lbs. or less) yet capable of carrying a modest payload (e.g., about 4 lbs. or more) that includes the array of sensors and onboard control systems discussed above with respect to sensor system 100. Figure 2 shows one generation of such vehicles with heterogeneous sensor modalities and constrained onboard computation power, including onboard navigational sensors, cameras, a laser scanner, and a computer. Multiple such low-cost and small UAVs 200 may be deployed in a farm for rapid tree mapping.

To exhibit the effectiveness of the disclosed systems and methods, several field experiments were conducted at a vineyard in Gait, CA, where a UAV 200 navigated itself around the environment, and a 3-D map of vines was reconstructed and visualized in real-time (See, Figures 3a and 3b). Figures 4a through 4c show sensor system 100 being autonomously flown above the test vineyard. Results from the experiments at this vineyard are presented in Figures 6a and 6b.

Alternatively or in addition, the present systems and methods may similarly be implemented using ground-based deployment systems 300, such as on unmanned ground vehicles (UGVs), on manned vehicles (e.g. a scout truck), or in a wearable or hand-held configuration of sensor system 100 provided on a human scout since human scouts periodically carry out inspections of farms. In particular, with respect to a configuration of ground deployment system 300 using human scouts, the present systems and methods may be deployed on a mechanically stabilized harness 310 that may be carried by such scouts. Mechanical stabilization improves the quality of the recorded data, resulting in higher precision of 3-D reconstruction. Being lightweight and self-contained, no additional setup is necessary for the human scout to walk around with sensor system 100 carried on harness 310 to him as demonstrated in Figures 4d and 4e. In some embodiments, a visualization device (e.g., a display attached to harness 310 or a wearable display provided with the scout, such as augmented reality (AR) goggles or the like) may provide the scout with real time reconstructed maps and other information for situational awareness.

Regardless of the particular implementation, ground-based deployment systems 300 may be used to acquire data of side views of trees. In certain scenarios, the data acquired by such ground-based deployment systems 300 may be used to aid the planning of flights by UAVs 200 or further ground-based data collection. In this way, multiple deployment mechanisms may be used in cooperation to generate the high-resolution 3-D model of the subject agricultural area. The data acquired by sensor systems 100 discussed above may be processed in multiple stages. Figure 5 shows an exemplary data processing pipeline, generally designated 400, of the present systems and methods. A middleware platform for robotic systems, such as the Robot Operating System (ROS), may form the backbone of data processing pipeline 400, facilitating sensor data logging and sharing of information between various processing nodes. State-estimation and mapping algorithms may be used to generate high-resolution multi-spectral 3-D maps, from which actionable intelligence such as fruit count, fruit size, trunk size, and canopy volume may be extracted using statistical machine learning techniques.

In some embodiments, for example, data processing pipeline 400 includes the generation of navigation data by a navigation sensor array 410 (e.g., navigational sensor 107, IMU 108, and stereo camera rig 105). A mathematical model may be used to estimate the location of the robot(s) (i.e., to determine the geographic location of a robot or sensor package with high accuracy). In some embodiments, for example, the data from the navigation sensor array 410 may be used by a state estimator 415 (e.g., an extended Kaiman filter (EKF)) to generate pose estimates for every science sensor on the platform. For example, the error-state formulation of the EKF may be employed. Further inputs are provided from a science sensor array 420 and a pose memory 430 that stores information regarding sensor relative poses. A point cloud assembler 440 may use the pose estimates received from state estimator 415, the science data from science sensor array 420, and the known relative poses between sensors from pose memory 430 to reconstruct a multi-spectral 3-D point cloud (i.e., a representation format for 3-D data of an environment). The point clouds may also be converted to an octree representation for efficient storage and analysis.

Furthermore, the reconstructed multi-spectral point cloud may be input to a predictive model 450 to extract useful, actionable information from the point cloud. In this regard, machine learning techniques may include a set of techniques to determine trends in data and to learn models for predicting properties of interest from observed data correlated to the desired properties of interest (e.g., crop yield, stress, or disease). This includes techniques for dimensionality reduction, unsupervised learning, and supervised learning.

In addition, the present systems and methods may be used for simultaneous localization and mapping (SLAM), wherein an unknown environment is explored, the location of the robot with respect to the environment is determined, and a map of the environment is build simultaneously. SLAM may be performed in 2-D or 3-D depending on the sensors used.

In some embodiments, the sensing modalities of the sensor suite are selected to monitor a range of plant physiological and morphological properties, and actionable intelligence can be extracted from the data acquired by the system. Specifically, with respect to farm management, four examples of data products that may be obtained using the systems and methods discussed above include reconstructing plant morphology, computation of plant vigor, estimation of leaf area, and automated fruit counting or yield estimation using remotely sensed data. Plant vigor, measured through NDVI, facilitates decision-making for fertilization. Accurate estimation of leaf area has the potential to improve pruning and spraying management. The capability to estimate yield accurately will enable growers to plan labor for harvesting and storage for harvested fruits, both of which may be facilitated by predictive models that use the acquired data to estimate a property of interest (i.e., leaf area or fruit count).

In particular, with respect to plant morphology, Figures 6a and 6b show a reconstructed point cloud of a row of grape trees from a sample vineyard. In Figure 6b, features of the environment (e.g., the canopy, trunk, ground) have been highlighted. In these images, data was collected with sensor system 100 facing the side of the grape trees.

Similarly, Figures 7a-9d show example data products obtained from a

3-D reconstruction of scans of a row of semi-dwarf apple trees across two trials. In particular, Figure 7a illustrates a multi-spectral 3-D reconstruction of a row of dwarf apple trees just using laser data, and Figure 7b illustrates a multi-spectral 3-D reconstruction of a row of dwarf apple trees with thermal data overlaid on a 3-D point cloud. The data was acquired in the afternoon, with the direction of the trees facing the sun showing higher canopy temperature (blue is cooler, red is warmer).

In addition, regarding plant vigor, in some embodiments, NDVI may be computed using multi-spectral imagery acquired by sensor system 100. For pixel data corresponding to a multi-spectral image, NDVI = (NIR - VIS) / (NIR + VIS), where NIR = 800nm (i.e., near-infrared) and VIS = 670 nm (i.e., visible). An NDVI map may then be generated using multispectral data acquired by sensor system 100 (e.g., onboard UAV 200). These maps enable growers to plan fertilization and mitigation in response to stresses observed in the NDVI imagery.

Regarding leaf area estimation, estimation of the total leaf area of a plant may help guide the management of fertilization, pruning, and spraying. In some embodiments, for example, the data from sensor system 100 is used to obtain accurate leaf area for target trees. Figures 8a-8d illustrate 3-D point clouds of canopies that may be used to determine tree height, canopy volume, and leaf area. In particular, Figures 8a and 8b show 3-D reconstructions of two apple trees from an orchard in Biglerville, Pennsylvania. Figures 8c and 8d show the representation of the trees in a format called Octree' that makes analysis and storage simpler. Here, an occupied voxel has an occupancy probability greater than 0.5, and volume of the tree was taken to be the bounding box of all occupied voxels. Weighting of the voxels was done using the occupancy probability. This metric may be referred to as the LiDAR area index. Figure 9 shows the correlation between the LiDAR area index and the true leaf area for each of the data points. This choice of input feature (i.e. LiDAR area index) shows a strong correlation, with an R-squared value of 0.82. The result demonstrates the use of sensor system 100 in estimation of leaf area for rows of trees. In this way, the leaf area estimation methodology may be used to rapidly estimate the leaf area of trees in a farm, enabling precise fertilization, spraying, and pruning.

Finally, accurate automated fruit counting can enable growers to determine storage and labor needs prior to harvest. Accordingly, in some embodiments, data acquired from sensor system 100 may be used to generate fruit count to provide this capability. In particular, the present systems and methods may be used with an algorithm to generate fruit counts for the rows of trees using the data acquired by sensor system 100. Although discussed hereinbelow in the context of counting oranges, those having skill in the art will recognize that the present approach may be extended to counting of other fruits such as apples, peaches, as well as clustered fruits such as grapes and blueberries.

In some embodiments, a fruit counting approach consists of two steps: fruit detection followed by fruit tracking. First, fruit detection is carried using a support vector machine (SVM) classifier that uses different color spaces to classify each pixel in an image as originating from a fruit or not. A series of images were labeled to annotate regions that have a fruit enclosed (e.g., an orange in a use case involving an orange orchard). The training dataset of images were used to train the SVM classier with candidate pixel colorspace values as input. In some embodiments, for estimating fruit yield from farms, a running count is generated as sensor system 100 carries out scans of trees. To do so, a fruit tracking algorithm may be used to track fruits detected in a stream of images. To keep track of fruits that have already been detected in previous frames, the optical flow of image descriptors may be computed across successive frames to estimate camera motion. The fruit tracking algorithm uses the estimated camera motion between frames to predict the locations of fruits detected in previous frames. These detections are compared with fruits detected in current frame to ensure previously detected fruits are not recounted.

A limitation of such an automated fruit counting algorithm is its dependence on suitable illumination. For example, fruit detection quality is poor when the sun faces the camera. Additionally, color based fruit classifiers are susceptible to false positives due to canopy features that sometimes match fruit features. For example, detection and counting of green oranges is challenging due to the color similarities with canopy. To address this issue, thermal imagery may be sued along with visible camera data to exploit the temperature differential between fruits and canopy in order to improve detection accuracy. Figures 10a and 10b illustrate data from thermal camera along with data from visible-range cameras that make detection of fruits efficient under appropriate conditions. Image processing algorithms are used to detect fruits and generate a running count as the sensor suite is deployed onboard UAVs or carried by a human scout.

The present subject matter is different from prior work in systems and methods for monitoring agricultural products (e.g., U.S. Patent Application Pub. No. 2013/0325346) in at least the following aspects. Compared to existing technologies, the present sensing system can be achieved at low- cost (e.g., about $10k or less), and it can be lightweight (e.g., about 4 lbs. or less), portable, compact (e.g., size of a show box), and self-contained (e.g., comes with onboard computing and batteries). In addition, it may be deployed in a distributed array, such as on affordable multi-rotor UAVs, including on a plurality of UAVs controlled as a swarm to collectively obtain the high-resolution 3-D imagery of a subject agricultural area. Alternatively or in addition, the present systems may be deployed by a human scout using a harness or by any of a variety of other mobile deployment device. Furthermore, the process for generating actionable intelligence may be addressed using machine learning techniques. Finally, the framework underlying the present subject matter considers data visualization as a valuable component of the technology stack.

The present subject matter can be embodied in other forms without departure from the spirit and essential characteristics thereof. The embodiments described therefore are to be considered in all respects as illustrative and not restrictive. Although the present subject matter has been described in terms of certain preferred embodiments, other embodiments that are apparent to those of ordinary skill in the art are also within the scope of the present subject matter.