for

SEA LICE DETECTION AND CLASSIFICATION IN AN AQUACULTURE

ENVIRONMENT

TECHNICAL FIELD

[0001] The present disclosure relates to computer vision techniques for detecting and classifying sea lice on freely swimming fish in an aquaculture environment.

BACKGROUND

[0002] The growth rate of world human population is applying substantial pressure on the planet’s natural food resources. Aquaculture will play a significant part in feeding this growing human population.

[0003] Aquaculture is the farming of aquatic organisms including fish in both coastal and inland areas involving interventions in the rearing process to enhance production.

Aquaculture has experienced dramatic growth in recent years. The United Nations Food and Agriculture Organization estimates that aquaculture now accounts for half of the world’s fish that is used for food.

[0004] Fish farm production technology is underdeveloped, when compared to the state of the art in other food production processes. Techniques that improve the production processes in fish farms using new perception and prediction techniques would be appreciated by fish farmers.

[0005] Sea lice on sea salmon and sea trout farmed in the open ocean represent one of the biggest challenges to fish health in the aquaculture industry. Sea lice is a natural parasite of salmon, trout, and other fish in saltwater, and are found in sea waters of the northern hemisphere. Sea lice damage fish by feeding on their mucous, skin, and blood. If sea lice become too prevalent in a fish farm, they can cause sores and other damage to the fish, laying them open to infection.

[0006] As a matter of food safety, governmental regulations covering aquaculture may impose an average per-fish limit on the number of sea lice. For example, Norwegian regulations specify for certain circumstances that there must be fewer than 0.5 adult female lice per fish on average in an aquaculture facility. The limit can vary depending on the season and geographic location of the farm. The fish farmer is responsible for keeping the numbers below the limit. Non-compliance may result in forced destruction of the fish stock.

[0007] In brief, WO 2017068127 describes a system that uses a track to force fish into a position next to the camera where it can be analyzed by a camera. The system is built so that fish are broken up into single file lines. The fish then move through the trough as the camera inputs the visual data into a computer. The computer then analyzes the data and determines if there are lice on the fish. The software also counts individual lice on the fish. The computer connects to an output device that displays the data.

[0008] Traditional techniques to count the number of sea lice are manual. However, minimizing the handling of fish is highly desirable not just because it is human-labor intensive but also because it impacts the health of the fish. As such, non-manual techniques are preferred.

[0009] The present invention disclosed herein address these and other issues.

[0010] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by their inclusion in this section.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings :

[0012] FIG. 1 is a schematic diagram of a sea lice detection and classification system.

[0013] FIG. 2 shows the different life stages and the life cycle of a particular sea lice species, the lepeophtheirus salmoms.

[0014] FIG. 3 is a flowchart of the high-level operations performed by the image processing sub-system of the sea lice detection and classification system.

[0015] FIG. 4 is a block diagram of the image processing sub-system of the sea lice detection and classification system.

[0016] FIG. 5 is a block diagram of a computer system with which the present invention may be implemented.

DETAILED DESCRIPTION

[0017] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

GENERAL OVERVIEW

[0018] Fish farmers would appreciate technology that can automatically provide them an accurate count of average sea lice per fish in a fish farm, including an accurate count of sea lice by gender and at different life stages. Such information could be used by fish farmers to determine compliance with regulatory requirements, assess the efficacy of sea lice control approaches, and efficiently use medicine for controlling sea lice. As mentioned, current methods for counting sea lice may be stressful to the fish. In addition, current methods may be inaccurate.

[0019] The deficiencies and other problems associated with current methods of sea lice detection and classification are reduced or eliminated with a method and system to automatically count the average number of sea lice per fish using a non-invasive, camera and computer vision system and method. The camera system is immersed in a net fish pen and captures images of freely moving fish from a substantially lateral perspective. The computer vision system automatically detects and classifies sea lice on the fish in the images. The average number of sea lice per fish by gender and / or at different life stages such as, for example, the average number of adult female sea lice per fish are computed based on the results of the detection and classification. The system and method have the advantage of being more automated and highly accurate and providing greatly reduced stress levels in fish, compared with current sea lice detection and classification techniques.

[0020] In addition to or as an alternative to determining the average number of adult female sea lice per fish, the average number of sea lice per fish at different life stages of sea lice is determined. The different life stage may include, for example, those depicted in FIG. 2. For example, the computer vision system may automatically detect and classify sea lice on fish in the images. From this, the average number of sea lice in, for example, the chalimus or the chalimus II life stage per fish may be computed based on the results of the detection and classification.

SEA LICE DETECTION AND CLASSIFICATION SYSTEM

[0021] FIG. 1 is a schematic diagram of sea lice detection and classification system 100 for detecting and classifying sea lice on freely swimming fish 102 in net pen 104. The system 100 includes pair of high-resolution, light sensitive, digital cameras 106 within a waterproof housing immersed underwater in net pen 104. While cameras 106 may be implemented by a stereo camera system, one skilled in the art will recognize that certain techniques disclosed herein for sea lice detection and classification can be used with a single lens system camera and do not require a stereo camera.

[0022] Each of cameras 106 is an approximately 12-megapixel monochrome or color camera with a resolution of approximately 4096 pixels by 3000 pixels, and a frame rate of approximately 60 frames per second. Although different cameras with different capabilities including color cameras may be used according to the requirements of the particular implementation at hand. For example, a color camera may be used to capture images that may be processed for sea lice detection and classification as well as for biomass estimation.

[0023] Selection of the camera lens for cameras 106 may be based on an appropriate baseline and focal length to capture images of a fish swimming in front of a camera where the fish is close enough to the lenses for proper pixel resolution and feature detection in the captured image, but far enough away from the lenses such that the fish can fit in both the left and right frames. For example, 8-millimeter focal length lenses with high line pair count (lp/mm) can be used such that each of the pixels in the left and right images can be resolved. The baseline of the cameras 106 may have greater variance such as, for example, within the range of 6 to 12-centimeter baseline.

[0024] Net pen 104 may be framed by a plastic or steel cage that provides a substantially inverted conical, circular, or rectangular cage, or cage of other desired dimensions. Net pen 104 may hold a number of fish 102 of a particular type (e.g., salmon) depending on various factors such as the size of net pen 104 and the maximum stocking density of the particular fish caged. For example, a net pen for salmon may be 50 meters in diameter, 20-50 meters deep, and hold up to approximately 200,000 salmon assuming a maximum stocking density of 10 to 25 kg/m3.

[0025] The techniques for sea lice detection and classification disclosed herein may be applied to a sea-pen environment such as net pen 104. However, the techniques may be applied to other fish farming enclosures. For example, the techniques may be applied to fish farm ponds, tanks, or other like fish farm enclosures.

[0026] Cameras 106 may be attached to a winch system that allows cameras 106 to be relocated underwater in net pen 104 so as to capture stereo images of fish from different locations within net pen 104. For example, the winch system may allow cameras 106 to move around the perimeter and the interior of net pen 104 and at various depths within net pen 104 to capture images of sea lice on both lateral sides of fish 102. The winch system may also allow control of pan and tilt of cameras 106.

[0027] The winch system may be operated manually by a human controller such as, for example, by directing user input to an above-water surface winch control system.

Alternatively, the winch system may operate autonomously according to a winch control program configured to adjust the location of cameras 106 within net pen 104, for example, in terms of location on the perimeter of the cage and depth within net pen 104.

[0028] The autonomous winch control system may adjust the location of cameras 106 according to a series of predefined or pre-programmed adjustments and /or according to detected signals in net pen 104 that indicate better or more optimal locations for capturing images of fish 102 relative a current position and/or orientation of cameras 106. A variety of signals may be used such as, for example, machine learning and computer visions techniques applied to images captured by cameras 106 to detect schools or clusters of fish currently distant from cameras 106 such that a location that is closer to the school or cluster can be determined and the location, tilt, and / or pan of cameras 106 adjusted to capture more suitable images of the fish. The same techniques may be used to automatically determine that cameras 106 should remain or linger in a current location and /or orientation because cameras 106 are currently in a good position to capture suitable images of fish 102 for sea lice detection and classification or other purposes.

[0029] It is also possible to illuminate fish 102 in net pen 104 with ambient lighting in the blue-green spectrum (450 nanometers to 570 nanometers). This may be useful to increase the length of the daily sample period during which useful images of fish 102 in net pen 104 may be captured. For example, depending on the current season (e.g., winter), time of day (e.g., sunrise or sunset), and latitude of net pen 104, only a few hours during the middle of the day may be suitable for capturing useful images without using ambient lighting. This daily period may be extended with ambient lighting.

[0030] Net pen 104 may be configured with a wireless cage access point 108 A for transmitting stereo images captured by cameras 106 and other information wirelessly to barge 110 or other water vessel that is also configured with wireless access point 108B. Barge 110 may be where on-site fish farming process control, production, and planning activities are conducted. Barge 110 may house computer image processing system 112 of the biomass estimation system 100.

[0031] Computer image processing system 112 generates accurate average sea lice per fish estimates on fish 102 in net pen 104 using computer vision and artificial intelligence techniques applied to the stereo images captured by cameras 106. Components and operation of image processing system 112 are discussed in greater detail below.

[0032] Cameras 106 may be communicatively coupled to image processing system 112 wirelessly via wireless access points 108. In addition, or alternatively, cameras 106 may be communicatively coupled to image processing system 112 by wire such as, for example, via a wired fiber connection between net pen 104 and barge 110.

[0033] Some or all of image processing system 112 may be located remotely from cameras 106 and may be operatively coupled to cameras 106 by wire or wirelessly. However, some or all of image processing system 112 may be a component of cameras 106. In this configuration, cameras 106 may be configured within an on-board graphics processing unit (GPU) or other on-board processor or processors capable of executing image processing system 112, or portions thereof.

[0034] In either configuration, the output of image processing system 112 including average sea lice per fish estimates may be uploaded to the cloud or otherwise over the Internet via a cellular data network, satellite data network, or other suitable data network to an online service configured to provide the estimates or information derived by the online service therefrom in a web dashboard or the like (e.g., in a web browser, a mobile application, a client application, or other graphical user interface.)

[0035] One skilled in the art will recognize from the foregoing description that there is no requirement that image processing system 112 be contained on barge 110 or that barge 110 be present in the aquaculture environment. Instead, cameras 106 may contain image processing system 112 or be coupled by wire to a computer system that contains image processing system 112. The computer system may be affixed above the water surface to net pen 104 and may include wireless data communications capabilities for transmitting and receiving information over a data network (e.g., the Internet).

SEA LICE LIFE STAGES

[0036] FIG. 2 depicts various life stages and life cycle of the lepeophtheirus salmonis species of sea lice. The techniques disclosed herein are applied to detect and classify the lepeophtheirus salmonis species of sea lice by gender and in different life stages. However, the techniques can be applied analogously to other species of the lepeophtheirus such as, for example, lepeophtheirus pectoralis. The techniques can be applied analogously to other types of fish (e.g., flatfish, European flounder, etc.) and other genera of sea lice (e.g., caligus) and other species within a genus of sea lice (e.g., caligus elongates.)

[0037] The general stages of the lepeophtheirus salmonis species from earliest to latest are: nauplius, copepodid, chalimus, pre-adult, and adult. During the nauplius stages, sea lice are small free-floating larvae approximately 0.7 millimeters in length and do not attach to fish.

[0038] During the copepodid stage, the sea lice attach to fish and become parasites. Their size during the copepodid stage is approximately 0.7 to 0.8 millimeters in length.

[0039] During the chalimus stages, the sea lice generally remain attached to the fish. During the chalimus stages, the sea lice may grow between approximately 1.1 millimeters to 2.3 millimeters in length. [0040] During the pre-adult phases, male and female sea lice are distinguishable by their respective shapes, with males having a generally barrel shape outline when viewed top-down and females having a generally triangular shape outline from the same perspective. As pre adults, male and female sea lice have an approximately mean length of 3.6 millimeters.

[0041] During the adult stage, the female sea lice are fertile. As mentioned, governmental regulations in some countries (e.g., Norway) put legal limits on the average number of sea lice per fish in a fish farm. For example, in Norway, the legal limit is set to 0.5 adult females per fish on average, without or without egg strings. During some parts of the year depending on the latitudinal location of the fish farm, the legal limit is lowered to 0.2.

[0042] The size in length of adult female sea lice generally ranges between 8 and 12 millimeters or more including egg strings. Adult females are visibly distinguishable from adult males and pre-adults by their egg strings or their total lateral length. Adult males are typically between 5 to 6 millimeters in length while adult females are typically between 8 to 12 millimeters in length or more.

IMAGE PROCESSING METHOD

[0043] FIG. 3 illustrates high-level process 300 performed by the image processing system 112 for sea lice detection and classification given an image captured by cameras 106. Note that if a stereo camera system is used, the given image can be one of the two stereo images captured in a pair (e.g., the base image).

[0044] At operation 302, fish in the given image are detected. The result of this operation is a bounding box that contains the pixels in the given image that depict a fish. Techniques for fish detection are described below.

[0045] At operation 304, the given image is cropped into rectangular sub-images where each sub-image corresponds to a bounding box or segmentation mask from operation 302 for a detected fish.

[0046] At operation 306, sea lice in the sub-images are detected and classified.

Techniques for detecting and classifying the sea lice are described below. At least two classes of detected sea lice may be tracked: (1) all detected sea lice, and (2) detected sea lice that are classified as adult females. Additional classes may be tracked including each of the different life stages of the sea lice (e.g., the life stage depicted in FIG. 2). For life stages in which gender is distinguishable (e.g., pre-adult and adult for 1. salmonis), such tracking may be done irrespective of gender (e.g., average number of pre-adult or adult sea lice per fish) or by gender (e.g., average number of pre-adult or adult female sea lice per fish). [0047] The average number of sea lice per fish and the average number of sea lice per fish in a tracked class (e.g., adult female) may be computed. The average number of sea lice per fish in the given image may be calculated based on the total number of sea lice detected in the given image at operation 306 and the total number of fish detected in the given image at operation 302.

[0048] The average number of sea lice per fish in the tracked class in the given image may be calculated based on the total number of sea lice detected in the given image that are classified as belonging to the tracked class at operation 306 and the total number of fish detected in the given image at operation 302.

[0049] These averages may be combined with the averages computed for other images captured by camera 106 over time and processed by system 112. In this way, the average sea lice population in net pen 104 over a period of time (e.g., 48 hours) including the average sea lice population in the tracked class (e.g., adult females) over the period of time is automatically determined. The length of the period of time may be a selected based on a variety of factors including the particular life stage or stages tracked, the lifecycle length of the sea lice, the time of year, how fast the sea lice grow or transition through a tracked life stage, etc.

IMAGE PROCESSING SYSTEM

[0050] FIG. 4 is a schematic diagram showing high-level components of image processing system 112. System 112 includes image storage 410, image filtration system 412, fish detection system 414, image cropping system 416, sea lice detection system 418, sea lice classification system 420, optional stereo matching system 422, and sea lice counting system 424.

[0051] At a high-level, operation of system 112 may be as follows. Images captured by camera 106 in net pen 104 are transmitted to image processing system 112 via a

communication link established between wireless access points 108A and 108B. The images received by the image processing system 112 are stored in image storage 410. Images are output from image storage 410 and input to image filtration system 412. In the case a stereo camera system is used in net pen 104, then pairs of rectified stereo images are transmitted to system 112 for storage in image storage 410.

[0052] Image filtration system 412 analyzes an input image to make a preliminary, relatively low-processing cost determination of whether the image contains suitable images of fish. If so, then the image is input to fish detection system 414 for further processing. In the case a stereo camera system is used in the net pen 104, then the filtration system 412 may analyze one of the two images of a stereo pair for suitability and, if suitable, pass the image on as input to fish detection system 414 for further processing. In addition, the pair of stereo images are also passed on as input to stereo matching system 422.

[0053] Fish detection system 414 identifies the pixels in an input image that correspond to a fish. The fish detection system 414 may be a bounding box for each fish detected in the image. The fish detection system 414 may output multiple bounding boxes for the same input image for different fish detected in the image.

[0054] Image cropping system 416 crops an input image in which fish detection system 414 detected one or more fish into sub-images. The cropping of the image in one or more sub-images corresponds to the bounding boxes around each detected fish in the input image. Each sub-image is input to sea lice detection system 418 for further processing.

[0055] Before inputting each sub-image to sea lice detection system 418, it is also possible to segment each sub-image corresponding to different parts of the fish body where sea lice are most likely to be found. This segmentation may be accomplished using convolution neural network-based image segmentation techniques. For example, sea lice typically attach to fish on the head, back, and perianal areas. According to some

embodiments, convolutional neural network-based semantic segmentation is performed on each cropped sub-image of fish to pixel segment out various parts of the fish body such as the head, back, and perianal areas where sea lice are most likely to be found. This segmentation can provide at least two benefits. First, an object that is detected in one of these segmented images areas may receive a numerical confidence boost that the object should be detected as or classified as sea lice, including classified as a particular life stage, based on the object’s presence in a segmented area of the image corresponding to a part of a fish body where sea lice are more likely to be present. Another benefit is this body part segmentation is that detection of sea lice in an image can be limited to just the pixels within the segmented areas so as to reduce the likelihood of false positive identification that might occur with respect to objects in areas of the image that do not correspond to one of the segmented areas.

[0056] Sea lice detection system 418 identifies candidate pixel regions in an input sub image, or a segmented portion of an image, that potentially correspond to a sea louse. A candidate pixel region may be designated by a bounding box the encloses the image region. Each candidate pixel region is input to sea lice classification system 420 to classify the candidate pixel region as sea lice or not sea lice and if sea lice, what class or life stage of sea lice (e.g., adult female sea lice or other).

[0057] Optional stereo matching system 416 detects pairs of corresponding pixels in a rectified stereo image pair. From this, stereo matching 417 outputs a disparity map or other disparity map-based information (e.g., a depth map) that can be used by classification system 420 to determine 3-D world coordinates for pixels corresponding to candidate pixel regions preliminary classified as sea lice so as to determine the length of the louse. The computed length may be used to classify the detected louse into one of the life stages described above. For example, a detected louse having a length greater than a threshold (e.g., 7 millimeters) may be classified as an adult female.

[0058] A disparity map or a depth map derived therefrom may be used to determine the real-world dimensions of the image region within a candidate bounding box to determine whether the image region within the bounding box should undergo further analysis. For example, the real-world dimensions of the image region as indicated by the disparity map or depth map may be such that the image object could not possibly be a sea louse. For example, if the image region bounded by the bounding box is greater than 5 centimeters in real-world length and /or width, then further analysis of the image region such as by classification system 420 may be not be performed (e.g., the image region may be discarded from further analysis) based on the expected maximum length and /or width of a sea louse (e.g., two centimeters).

[0059] Image filtration system 412 may include a convolutional neural network (CNN) computer system for analyzing an input image to perform classification on objects detected in the image. The CNN system, as well as other CNN systems used in system 112, may be composed of an input layer, an output layer, and multiple hidden layers including convolutional layers that emulate in a computer the response of neurons in animals to visual stimuli.

[0060] Given an input image, the CNN system of the image filtration system 412 may convolve the image by a number of convolution filters. Non-linear transformations (e.g., MaxPool and RELU) may then then applied to the output of convolution. The CNN system may perform the convolution and the non-linear transformations multiple times. The output of the final layer may then be sent to a softmax layer that gives the probability of the image being of a particular class (e.g., an image of one or more fish). Alternatively, the output of the final layer may be sent to a sigmoid function. Image filtration system 412 may discard the input image from further processing by system 112 if the probability that one or both of the stereo images is not an image of one or more fish is above a threshold (e.g., 80%). Instead of binary classification (e.g., fish / not fish), it is also possible to use a softmax layer or a sigmoid function for multi-class classification such as, for example, four classes consisting of: unknown, not fish, clear image of fish, blurry image of fish. The appropriate downstream processing of an image may be performed depending on the classification of the image. For example, images classified as not fish may be discarded from further processing by system 112, while images classified as clear images may be passed on for further processing by system 112, and images classified as blurry image may undergo an image sharpness correction process before being passed on for further processing by system 112.

[0061] Assuming an input image is not discarded by image filtration system 412, the image be input to fish detection system 414.

[0062] Fish detection system 414 accepts an image passed through by image filtration system 412 and uses a convolutional neural network system to identify fish in the image via a bounding box. A bounding box may also be associated a label identified the fish within the bounding box. This results in a set of region proposals which are input to the CNN system trained on images of fish with the locations of the fish within the fish labeled to determine which regions contain fish and which do not.

[0063] The images of fish on which the CNN system is trained may be representative of images containing fish from which sea lice can be detected and classified by sea lice detection system 418 and sea lice classification system 420. For example, the CNN system may be trained on images of fish that provide a lateral view of the fish including the head and tail at various different yaw, roll, and pitch angles and at different sizes in the image representing different distances from the camera 106.

[0064] In some embodiments, such training data is generated synthetically using a computer graphics application (e.g., a video gaming engine) in order to generate sufficient accurate training data.

[0065] A final layer of the CNN system may include a support vector machine (SVM) that classifies the fish in each valid region. Such classification may include a fish is detected in the region. Fish detection system 414 may tighten the bounding box around a region of a fish by running a linear regression on the region. This produces new bounding box coordinates for the fish in the region.

[0066] Fish detection system 414 may discard an input image from further processing by system 112 if no fish are detected in the image. [0067] To speed up fish detection, it is also possible to use convolution and pooling (e.g., region of interest pooling) to leverage computation performed during the forward pass of the CNN across region proposals.

[0068] In addition, or alternatively, the CNN, the classifier, and the bounding box linear regressor may be jointly trained for greater accuracy. This may be accomplished by replacing the SVM classifier with a softmax layer on top of the CNN to output a classification for a valid region. The linear regressor layer may be added in parallel to the softmax layer to output bounding box coordinates for the valid region.

[0069] To speed up fish detection, it is also possible to leverage the convolutional feature maps used by a region-based detector such as Fast R-CNN, to generate region proposals such as is done with Faster R-CNN.

[0070] The bounding box of a fish may be cropped by the image cropping system 416 from the image in which is detected.

[0071] The CNN-based object detection may then be performed again on the cropped image by sea lice detection system 418 to detect and obtain bounding boxes around sea lice on the fish in the cropped image. For this, a separately trained CNN may be used. The separate CNN may be trained on tight images of fish (synthetically generated or images captured in situ from a camera) with the locations of sea lice in the images labeled in the tight images.

[0072] Sea lice detection system 418 may perform pixel level segmentation on the cropped image. This may be accomplished using an additional CNN system that runs in parallel with the object detection CNN system such as Mask R-CNN. The output of this are binary masks for the locations of the detected sea lice in the cropped image.

[0073] As an alternative to using a convolutional neural network, sea lice detection system 418 may detect regions of interest in cropped images of fish using a specularity detection method. Such a method can be effective because sea lice typically appear as bright specularities in the image. Some possible specularity detection methods that could be applied to detect sea lice in the cropped images are described the following papers:

[0074] Stephane Tchoulack, J.M. Pierre Langlois and Farida Cheriet, "A Video Stream Processor for Real-time Detection and Correction of Specular Reflections in Endoscopic Images,” Circuits and Systems and TAISA Conference on pp. 49-52, 2008; and

[0075] Francisco Ortiz and Fernando Torres,“Automatic Detection and Elimination of Specular Reflectance in Color Images by Means of MS Diagram and Vector Connected Filters,” IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS— PART C: APPLICATIONS AND REVIEWS, VOL. 36, NO. 5, on pp. 681-687, SEPTEMBER 2006.

[0076] Sea lice classification system 420 performs feature matching between an image of a sea louse such a cropped image of a fish in which a sea louse is detected or a candidate region of interest thereof. For example, the feature matching may be based on the“ORB” features as described in the 2011 paper by Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary R. Bradski,“ORB: An efficient alternative to SIFT or SURF.” The ORB features give a good estimate of the context / neighborhood of the sea louse.

[0077] The context / neighborhood of the sea louse is useful because the relatively small size of the louse in the image makes it difficult to extract meaningful features from the image of the louse itself. As such, the ORB features facilitate examination of the context / neighborhood where a louse is most likely to be present. As an alternative, one or more HAAR-like feature descriptors may be used. Since sea lice generally appear brighter than their surroundings, the HAAR-like feature descriptor may be useful to capture the intensity related patterns found in images of sea lice since the features capture the intensity differences in multiple spatial frequencies. ORB features and HAAR-like feature descriptors may also be combined.

[0078] Sea lice classification system 420 may include a trained support vector machine trained on feature descriptors labeled louse, not-louse, adult female sea louse, adult sea louse, or other different life stage classifications (e.g., two mating sea lice). A region of interest in an image may be classified as louse, not-louse, adult female sea louse or other life stage based on the feature descriptors obtained for the region of interest.

[0079] The SVM classifier may be trained on feature descriptor examples labeled only louse and non-louse. In this case, if a region of interest is classified as louse by the SVM classifier, then length information about the louse that may be computed using the disparity map produced by stereo matching system 422 may be used to further classify the louse as non-adult female and adult female.

[0080] Sea lice counting system 424 tracks for individual images processed by image processing system 112 and across images processed by system 112, the total number of fish detected, the total number of sea lice detected, and the total number of adult female sea lice detected. From these counters, the average number of sea lice per fish and the average number of adult female sea lice per fish can be computed.

FISH DETECTION [0081] A version of the FastMask image segmentation algorithm may be used to obtain a segmentation mask and / or a bounding box around a fish in an image or an image cropped therefrom. More information on the FastMask algorithm can be found in the paper by H. Hu, S. Lan, Y. Jiang, Z. Cao and F. sha,“FastMask: Segment Multi-Scale Candidates in One Shot,” arXiv: l6l2:08843v4, April 11, 2017.

[0082] The FastMask algorithm may miss (false negative) a fish that is near to the cameras 106 when the image is captured. This is problematic for the purpose of sea lice detection and classification because generally there are fewer false negatives detecting sea lice in images where the fish and any sea lice thereon are closer in distance to cameras 106 and thus represents a larger portion of the camera images, when compared to detecting sea lice in images where the fish and any sea lice thereon are farther away from the cameras 106.

[0083] A dense optic flow-based segmentation algorithm may be used for fish detection. One example of a suitable dense optic flow algorithm that may be used for fish detection is described in the paper by C. Lie, W. Freeman, E. Adelson,“Beyond pixels: exploring new representations and applications for motion analysis,” Massachusetts Institute of Technology, 2009.

[0084] To improve the processing time of the algorithm, an estimate of dense optic flow is obtained using downsampled images. For example, images obtained by the cameras 106 may be downsampled to one-fourth (¼) their original size.

[0085] Fish segmentations using a dense optic flow-based algorithm proceeds as follows. The dense optic flow system is trained on actual or synthetically generated video of freely swimming fish at a particular frame rate (e.g., 30 frames per second). The trained system is used to compute dense optic flow from video frames captured by cameras 106. For example, if cameras 106 capture video at 60 frames per second, then every three frames may be used by the training system to compute dense optic flow. The optic flow magnitude and angle images produced by the trained system from the captured video frames may then be clustered for approximate motion segmentation using a mean shift algorithm. For example, a suitable mean shift algorithm is described in the paper by D. Comaniciu and P. Meer,“Mean Shift: A robust approach toward feature space analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002. The clustered regions may then be filtered based on eccentricity and area to identify possible fish segmentations.

SEA LICE DETECTION

[0086] Sea lice detection may involve the following steps:

[0087] Step 1: selecting candidate points for sea lice classification, [0088] Step 2: annotating (labeling) for sea lice,

[0089] Step 3: building a classifier, and

[0090] Step 4: classification.

[0091] For candidate point selection, a threshold-based method may be used to generate binary masks, label regions, and select regions whose area (in pixels) is between 25 to 150 and have an eccentricity of 0.75, 1. The candidate point selection algorithm may be modified based on detected sea lice properties such as intensity, color and spatial frequency properties of known sea lice locations in images.

[0092] A specularity detection method may be used for candidate point selection to leverage the appearance of sea lice in images which typically appear as bright specularities. Some possible specularity detection methods that may be used is described in the following papers:

[0093] Stephane Tchoulack, J.M. Pierre Langlois and Farida Cheriet, "A Video Stream Processor for Real-time Detection and Correction of Specular Reflections in Endoscopic Images,” Circuits and Systems and TAISA Conference on pp. 49-52, 2008; and

[0094] Francisco Ortiz and Fernando Torres,“Automatic Detection and Elimination of Specular Reflectance in Color Images by Means of MS Diagram and Vector Connected Filters,” IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS— PART C: APPLICATIONS AND REVIEWS, VOL. 36, NO. 5, on pp. 681-687, SEPTEMBER 2006.

SEA LICE CLASSIFICATION

[0095] ORB features may be used to build a sea lice classifier. The ORB features provide a useful estimate of the neighborhood context of the sea lice because they capture orientation features well and extraction is computationally efficient. For example, the background is useful because sea lice are relatively small and extracting meaningful features from the pixels of the sea lice itself is relatively difficult, if not impractical. Instead, examining the image in the area of the where sea lice are most likely to be present as indicated by segmentation masks of the head, tail, and perianal area of the fish may be more useful.

[0096] In addition to or instead of ORB feature descriptions, HAAR features descriptors may be used. Sea lice generally appear in images to be brighter than their surroundings, HAAR features are useful for capturing the intensity differences at many spatial frequencies and thus are useful for capturing the intensity related patterns found on sea lice

SEA LICE DETECTION AND CLASSIFICATION [0097] An objective of sea lice detection and classification is to compute the ratio of the number of sea lice per fish. Achieving this objective may involve: (1) detecting the number of fish in an image, and (2) detecting sea lice present on fish, if any.

FISH DETECTION AND SEGMENTATION

[0098] An object of fish detection is to obtain a bounding box around a fish in an image. For example, the FaskMask image segmentation algorithm may be used for this. The results of FastMask are encouraging for the small fish, but the algorithm typically misses the large fish present in the image. For the purpose of sea lice detection, this is sub-optimal. Because, fish that appear small on the image are generally far away from the camera. Sea lice, being extremely small compared to the size of the fish, are seen more clearly in the fish that are very close to the camera. On the far away fish, sea lice are difficult to distinguish. Therefore, it is important to get good segmentation or bounding boxes for the large size that are near-to- camera fish.

[0099] One possible approach to accomplish this is to use dense optic flow-based segmentation using the Fameback method in OpenCV. Very good motion magnitude and orientation estimates can be obtained using the method of Sun, Deqing, Stefan Roth, and Michael i. Black. "Secrets of optical flow estimation and their principles." Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on IEEE, 2010. Various background subtraction methods available with OpenCV are also possible such as described in OpenCV background subtraction at

https://docs.opencv.Org/3.3.0/db/d5c/tutorial_py_bg_subtr action.html.

[0100] The PyFlow method may also be used for moving fish segmentation:

https://github.com/pathak22/pyflow. PyFlow gives a good estimate of dense optic flow compared to Farenback method. But, it also is also very time consuming. For a 1092x1080 image, it takes about 40 seconds. Since accurate segmentation is not necessary for fish detection, the image can be downsampled to l/4th size and then segmentation performed on the downsampled image. If the PyFLow is trained on 30 fps videos, then every 3rd frame for computing dense optic flow may be used, assuming video is captured at approximately 60 frames per second. Then, the optic flow magnitude and angle images are clustered for rough motion segmentation using the mean shift algorithm available from ski earn cluster. The clustered regions are filtered based on eccentricity and area to identify possible fish segmentations.

SEA LICE DETECTION [0101] At a high-level, sea lice detection may involve selecting candidate points for sea lice classification, annotating for sea-lice/ not sea-lice or by life stage, building classifier and finally classification.

[0102] For candidate point selection, a threshold-based method to generate binary masks may be used. The regions may then be labeled the regions and the regions selected whose area (in pixels) is between 25 to 150 and have an eccentricity of 0.75, 1.

[0103] Once the annotations for sea-lice are obtained, the intensity, color and spatial frequency-based properties of sea lice locations may be examined and the candidate point selection algorithm modified based on the observed sea lice properties. A simple threshold and binary morphology-based method may work well.

[0104] Since sea lice appear as bright specularities, specularity detection methods may be used for candidate point selection.

SEA LICE CLASSIFICATION

[0105] Feature descriptors can be useful in building the sea lice classifier. The ORB features give a good estimate of the context (“neighborhood”) of the sea lice. The background can be useful because the sea lice are extremely small and extracting any meaningful features from sea lice itself may not help much. Instead we should look for the surrounding context, i.e., the neighborhood in which sea lice are most likely to be present.

[0106] Since ORB feature descriptors capture the orientation features very well, and extremely fast, this is one candidate. Another candidate is the HAAR feature descriptor.

Since sea lice appear to be brighter than the surroundings, the HAAR descriptors which capture the intensity differences at many spatial frequencies can capture the intensity related patterns found on sea lice.

[0107] It is also possible to combine ORB and HAAR features.

SEA LICE CLASSIFICATION - FEATURES FROM VIDEO

[0108] Feature descriptors (e.g., ORB, HAAR and Local Binary) may be extracted from input training videos with annotations. Image patches (RGB and gray) may be extracted, rotated image patches based on the annotation (RGB and gray) and feature descriptors around locations annotated as‘lice’ or‘non lice’. The extracted features may be stored for analysis or training an SVM classifier.

SEA LICE CLASSIFICATION - SUPPORT VECTOR MACHINE TRAINING

[0109] Feature descriptors for‘lice’ and‘non lice’ classes are made available. An SVM classifier is then trained. Multi -resolution grid search with 10-fold cross-validation is used for training the ORB feature descriptor based SVM classifier. Radial Basis Function kernels are used. The two hyper-parameters $C$ and $\gamma$ are tuned using a multi-res approach. The default grid can be obtained, and values changed using the following code. Only example for $C$ is below:

[0110] C grid = mdl.getDefaultGridPtr(cv2.ml.SVM_C)

[0111] C_grid.minVal=le-lO

[0112] C_grid.maxVal=lelO

[0113] C_grid.logStep=2

[0114] C grid.logStep is the step interval in which grid values are incremented. The grid steps are computed as follows:

[0115] [C grid.minVal, C_grid.minVal*C_grid.logStep,

C gri d. mi n V al * C gri d.1 ogS tep ^A 2. C_grid.minVal* C grid.logStep ⁷ 3. ... , C grid.maxVal | [0116] So, the grid resolution is controlled by C grid.logStep. The grid can be refined using this parameter based on desired accuracy and the accuracy obtained using the SVM classifier. svm.trainAuto function of OpenCV does the training where 10-fold cross- validation is carried out by the svmtrainAuto function, so no need to do additional cross- validation.

[0117] Depending on grid resolution, number of training examples, range of values (minVal and maxVal), training can take a lot of time. For the above values, it takes about 512 hours on Intel Core i7 machine with 8GB RAM.

[0118] Once training is completed, the trained SVM model is saved.

SEA LICE CLASSIFICATION - VALIDATION

[0119] The trained SVM classifier may be verified to determine if it is working as expected. Since OpenCV version of SVM classifier does not allow us to access the cross- validation score during training, this validation step is important. In this validation step, only the already annotated locations are classified using the trained model to compute the precision and recall metrics. In this validation step, the results are shown on each frame of video for better visualization. Each location may have two squares. The inner square represents the ground truth, based on the annotation. If the square is green, it denotes a lice location, red indicates a non-lice location. The outer square represents the classification result, with same color coding. So, if a location is correctly classified, you will see two squares of matching color at that location, irrespective of whether it is red (for non-lice) or green (for lice).

SEA LICE CLASSIFICATION - FINAL STEP [0120] This is the last step, where given a video, sea lice key points or possible sea lice candidate locations are automatically extracted, and the classification of each candidate location is done. If a candidate location is detected as sea lice, it will be visualized in the video with a blue square around that location. Sea lice key points are currently chosen based on a simple threshold. On the gray image, anything with a gray value above 100 is set to +1 in a binary mask. The binary mask in labeled for connected components, then each connected component which has eccentricity above 0.75 and area between 150 - 250 pixels is chosen as a candidate sea lice location. These parameters can be tuned. Also, a different method based on the properties of sea lice patches could be used in this step.

HARDWARE OVERVIEW

[0121] FIG. 5 is a block diagram that illustrates a computer system 500 with which the foregoing exemplary embodiments may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled with bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor, a central processing unit (CPU) or a core thereof, a graphics processing unit (GPU), or a system on a chip (SoC).

[0122] Computer system 500 also includes a main memory 506, typically implemented by one or more volatile memory devices, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 also may be used for storing temporary variables or other intermediate information during execution of instructions by processor 504. Computer system 500 may also include a read-only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, typically implemented by one or more non-volatile memory devices, is provided and coupled to bus 502 for storing information and instructions.

[0123] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display (LCD), a light emitting diode (LED) display, or a cathode ray tube (CRT), for displaying information to a computer user. Display 512 may be combined with a touch sensitive surface to form a touch screen display. The touch sensitive surface is an input device for communicating information including direction information and command selections to processor 504 and for controlling cursor movement on display 512 via touch input directed to the touch sensitive surface such by tactile or haptic contact with the touch sensitive surface by a user’s finger, fingers, or hand or by a hand-held stylus or pen. The touch sensitive surface may be implemented using a variety of different touch detection and location technologies including, for example, resistive, capacitive, surface acoustical wave (SAW), or infrared technology.

[0124] An input device 514, including alphanumeric and other keys, may be coupled to bus 502 for communicating information and command selections to processor 504.

[0125] Another type of user input device may be cursor control 516, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 504 and for controlling cursor movement on display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

[0126] Instructions, when stored in non-transitory storage media accessible to processor 504, such as, for example, main memory 506 or storage device 510, render computer system 500 into a special-purpose machine that is customized to perform the operations specified in the instructions. Alternatively, customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or hardware logic which in combination with the computer system causes or programs computer system 500 to be a special-purpose machine.

[0127] A computer-implemented process may be performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform the process.

[0128] The term“storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media (e.g., storage device 510) and/or volatile media (e.g., main memory 506). Non-volatile media includes, for example, read-only memory (e.g., EEPROM), flash memory (e.g., solid-state drives), magnetic storage devices (e.g., hard disk drives), and optical discs (e.g., CD-ROM). Volatile media includes, for example, random-access memory devices, dynamic random-access memory devices (e.g., DRAM) and static random-access memory devices (e.g., SRAM).

[0129] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the circuitry that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0130] Computer system 500 also includes a network interface 518 coupled to bus 502. Network interface 518 provides a two-way data communication coupling to a wired or wireless network link 520 that is connected to a local, cellular or mobile network 522. For example, communication interface 518 may be IEEE 802.3 wired“ethemet” card, an IEEE 802.11 wireless local area network (WLAN) card, a IEEE 802.15 wireless personal area network (e.g., Bluetooth) card or a cellular network (e.g., GSM, LTE, etc.) card to provide a data communication connection to a compatible wired or wireless network. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0131] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through network 522 to a local computer system 524 that is also connected to network 522 or to data communication equipment operated by a network access provider 526 such as, for example, an internet service provider or a cellular network provider. Network access provider 526 in turn provides data communication connectivity to another data communications network 528 (e.g., the internet). Networks 522 and 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are example forms of transmission media.

[0132] Computer system 500 can send messages and receive data, including program code, through the networks 522 and 528, network link 520 and communication interface 518. In the internet example, a remote computer system 530 might transmit a requested code for an application program through network 528, network 522 and communication interface 518. The received code may be executed by processor 504 as it is received, and/or stored in storage device 510, or other non-volatile storage for later execution.

[0133] In the foregoing specification, the embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.