Field

The present invention relates to a monitoring and recording system. More particularly the invention relates to a monitoring and recording system for monitoring items such as road signs, street lamps, traffic lights and street markings, all of which items are collectively known as, and hereinafter referred to as, items of street furniture or assets.

Background

There is a continuous need, especially by town councils, town planners, traffic departments, maintenance and repair companies and local authorities, to monitor, count and catalogue items of street furniture and their state of repair for accounting and maintenance purposes. This is particularly important for planning and when managing budgets for maintenance, repair and replacement.

Prior Art

A number of automatic surveillance systems exist in order to monitor traffic, especially vehicles that have been illegally parked. One example is that described in UK Patent Application GB-A-2 527 093 (TESCAP) which relates to a vehicle parking enforcement system.

The system described determines whether a parked vehicle is permitted to park in a particular location and automatically monitors the whereabouts of traffic enforcement officers carrying hand held communication devices; then allocates a suspected parking violation to the nearest traffic enforcement officer to investigate and issue a parking violation ticket as appropriate.

Another example of an automatic surveillance system is described in EP 3 379 459 (Tata Consultancy Services Limited), wherein a system and method for automated inventory management are provided. The method includes obtaining street-view images of a geographical area having telecom assets. The telecom assets are associated with corresponding GPS location coordinates. An object recognition model is applied to the street-view images to detect the telecom assets therein by way of asset labels. US 6266 442 (Facet Technology Corp) teaches an apparatus that analyses frames of digitised video data which may include objects of interest. There is a need to detect, classify and identify these objects by filtering the video data. The system, when used in conjunction with GPS, calculates object location, classifies each object by type, extracts legible text appearing on a surface of an object and stores a visual representation of the object in a form dictated by the user.

US 2012/0320204 (3M Innovative Properties) teaches an asset assessment system that gathers information about a plurality of assets located at various geographical locations.

US Patent US 2018/0060986 (Toshiba KK) discloses an imager that captures, from a vehicle, an image of the surroundings and stores image data and identification information for identifying a road structure contained in the image data.

International Patent Application WO 2016/112435 (Boyle) describes a method for roadside asset tracking and maintenance monitoring, comprising deploying a mobile unit alongside a roadside section. The mobile unit comprising a data capture device operably coupled to at least one digital camera; a GPS receiver; and a data interface for communicating with an asset management server.

The server includes an asset tracking register database. The method utilises asset type image recognition techniques for automating the identification of the roadside assets and comparing roadside asset data with an asset tracking register database to record newly identified roadside assets and their respective locations and identifies missing roadside assets.

US 2017/0287170 (California Institute of Technology) teaches a method for identifying geographic locations and for performing a fine-grained classification of elements detected in images which are captured from multiple viewpoints or perspectives. The method identifies locations by using a probabilistically technique of combining predictions from different viewpoints to a common geographic coordinate frame. Fine-grained image classification is performed by comparing image portions from several images associated with a particular geographic location. An aim of the present invention is to provide a system that is able to be used in real time in order to monitor the state of roadside assets, record their location and verify the existence and status of items of street furniture.

The invention arose in order to provide an improved system that is capable of being readily implemented and used alongside existing monitoring and recording systems and which reduces false labelling.

Summary of the Invention

According to a first aspect of the present invention there is provided a monitoring and recording system for compiling an asset register of items of street furniture comprising: at least a first camera imager and a LIDAR imager, in use the imagers are vehicle mounted, the LIDAR imager derives point cloud image data corresponding to image data obtained from items of street furniture that are also imaged by the at least first camera imager; a global positioning system (GPS) provides GPS location data which is associated with each frame of image data; a memory stores digitised image data; and a transmitter transmits stored digitised image data files and associated GPS location data to at least one database, wherein a processor processes the point cloud image data and identifies cloud image points that are collinear with an imaged item of street furniture and the first camera imager to derive perspective data; and the processor operates in accordance with image recognition and pattern recognition software (MASK RCNN) to: process the perspective data of the imaged item of street furniture and to interpret the perspective data and the digitised image data, thereby to identify an item of street furniture; to provide labelled image data of the item of street furniture; and to combine the labelled image data with the GPS location data in order to produce a unique identifier for an item of street furniture at a specific GPS location.

An advantage is that the system operates automatically by using image data derived from street furniture assets whose pixel widths and/or or heights are compared with assumptions of their dimensions to estimate a location for the asset. This removes the need for human input when estimating a location.

It is therefore apparent that the method deployed by the present invention to assess assets is superior to existing methods because it requires classification using only one image instead of using a plurality of images and uses a simple formula to estimate asset location which is quicker to compute and less prone to errors.

An example of a formula that is used is in order to derive distance from image data is: distance of object (mm) from imager = imager focal length (mm) x object real height (mm) x image height (pixels) ÷ object height in image (pixels) x sensor height (mm).

The formula refers to height but in practice any characteristic dimension, (diameter or width) may be used.

The system uses a single image to generate an asset registry entry or a single LIDAR “snapshot” to indicate the likely identity of an object. When both are combined this therefor acts as two-step verification process for asset classification. The system has been found to remove many false detections, such as incorrect classifications inside an image where no asset was actually present, that were often encountered in previous systems.

By processing 360° images in the survey in reverse order, that is by processing a final image before a penultimate image, the system effectively checks that the image of an asset seen first is the closest to the imager and so usually is the best image available in order to assess its identity. Optionally as in a real time situation there is no“start” or“end” at the end of a survey.

Optionally the LIDAR image data is used to derive a record of road condition as point cloud data can be analysed to determine depth of potholes and condition of road independently or in conjunction with a regular camera.

Some prior art systems identified geographic locations by using a probabilistically model combining predictions from different viewpoints by using outputs in a common coordinate frame. The present invention does not do this. Instead it performs location calculations based on an estimated size of a detected object that is derived from one image, from one of a plurality of available images, which are captured using a 360° camera. Preferably the system uses image data, in conjunction with knowledge of the focal length of the camera in order to determine a distance of an imaged object from the camera.

Furthermore by using the angle at which the asset is detected and imaged, with respect to a datum in the 360° cameras field of view, a directional bearing is derived at which the object is located with respect to one of the plurality of cameras. For example one camera may have an angular field of view of between 0° and 75° heading of a vehicle and software can be configured to anticipate this in advance so that vehicle heading from the GPS can be used in conjunction with the asset detection angle in order to work out the true direction at which the asset is detected relative to world/map north.

Another criteria is to assess image size and employ a preferred image as an image standard. As images obtained closets to an item of street furniture tend to have most pixels relating to that item the system is adapted to select and use the latest acquired image for image processing and identification purposes.

The system derives both distance data and direction data. This data is combined with recorded GPS location data and the distance data and direction data of the vehicle on which cameras are located, in order to provide a prediction of the absolute location of a specified asset. By employing data derived from the LIDAR, data points are detected at the same instant of capture as the 360° images. Software is configured to find and identify assets by identifying points in an image that are associated with the particular asset. Therefore by combining known positions of image points, using data from the LIDAR and the camera, it is possible to compensate for differences in perspective. This is the case because the camera and LIDAR are at the location, for example on the vehicle that is acquiring the image data.

The preferred technique used to estimate the location and identity of an asset is also faster than existing systems because it uses one captured image and so avoids the need to perform complex and time consuming computations in order to derive data from different viewpoints and in order to compensate for perspective changes which was the case in many existing systems. The invention achieves this by employing LIDAR derived data to add to the asset classification process by generating a two dimensions (2-D) perspective image of the LIDAR data which a processor operating in accordance with software is trained to recognise patterns relating to asset specific asset classifications. Therefore for example the LIDAR image data enables rapid differentiation between different types of asset by classifying various sub-types of asset as an example either wall mounted or post mounted sign.

By performing analysis using data derived from only forward facing cameras processing times are reduced and it has been found that the system achieves around a 3-fold increase in speed of asset identification and classification with the least possible assets being unseen during image capturing. This is because most, if not all, road viewable assets are seen by these cameras at some point in the survey; although as mentioned earlier most recent images are usually used to identify an image and its dimensions unless there is obscuring in which case the software is configured to continuously select a previously obtained image.

Optionally an optical character recognition means may be included in order to increase camera resolution and automatically record text on road signs.

In some embodiments post survey software permits a user to operate tools to“fix” 360° surveys and thereby remove any GPS drift issues that can occur as a result of signal pathway losses that may arise from reflections from tall buildings for example. Such post survey analysis enables the actual true location points to be verified and set as a datum.

In another preferred embodiment the invention is adapted to operate in combination with a signal received from a mobile communication device, such as‘smart phone’ which is operating application specific software (APP). When so configured such a mobile communication device permits users, who may be walking along an asset (for example a double yellow line indicating parking restrictions), with a camera or microphone for recording on the job notes about the asset and the location the assets is in by sampling the GPS as the user walks along the asset to create a“GPS path” or single GPS point to be combination to generate an asset. Such recorded image data or audio data can be reviewed or automatically accepted based on a customer preference and used to generate an asset database entry for the asset regardless of it coming from a survey vehicle, mobile telephone App or any other source, the system allows users to review or automatically submit assets to their register.

Optionally the data may be derived from crowd sourced data in order to provide “training images” for artificial intelligent (Al) image recognition software. Such Al image recognition software may be configured in order to enable an uploaded image taken using a mobile device to be used in order to perform an update or verification of an asset at a specified location.

In this sense users and operators may specify a specific function to be included in the Al image recognition software or the system as a whole in order to enable its customisation for use by a particular user group. For example a local council may request a heat map of an area in which a colour overlay is used to indicate proximity of assets. Such heat maps may also be used to identify zones of high pollution levels or litter and provide an alert to relevant authorities. Or, where parking fines have been disputed, the system may be configured to provide a report and verification of signage and a vehicle’s identity in order to evidence a penalty fine being issued.

In a preferred embodiment the system obtains image data from six separate imagers and stitches them together. However, image processing is performed on each independently from the others and so the invention relies on only one image to generate an asset entry, whereas some prior art systems rely heavily on multiple image detection of the same object in order to pinpoint its location which required more time and more image processing. The images are used to provide users with a 360° perspective image for them to view.

The system enables data to be editable, updatable and reviewable by authorised users. Optionally the software enables automatic blurring or obscuring of sensitive or private data from images, such as faces or vehicle licence plates. This is in part because all data points are reviewable so a user can search, correct or reject those using editing features of the software. Another advantage is that, due to the machine learning aspect, there is no need to align the imagers in a precise configuration to allow perspective calculations, as these are computed using point cloud data.

By “knitting” together images there is no need to perform time consuming estimations of an area, for example by way of triangulation techniques on a set of multi-view images selected from the street-view images, which was the case in some prior art systems.

Another advantage of the invention over existing systems is that because a 360° camera is used and images are ‘stitched’ the six captured images to create a panoramic image can be processed by the Al in parallel to speed up processing time.

The system supports both automated and manual surveying of assets while also enabling the user to modify and update assets within the system. Some existing systems lacked a user interface to facilitate data modification, data review, 360° street views based manual surveying, or physical surveying.

360° street view based surveying is the act of a user viewing a street view using the systems user interface and generating an single non 360° image from the 360° image of an asset and then manually labelling it and determining its location, optionally the system will offer likely classifications and locations for the asset they have cropped from the street view but these can be overrode if wanted.

In a preferred embodiment the invention can process the regular image to detect assets to generate an asset entry but also uses LIDAR data to get a 2nd classification which allows it to confirm/reject/verify assets (for example a LIDAR pattern which the Al sees commonly associated to certain assets) and this lowers Al false positives but also use it to discriminate between type of asset which fall into the same classification (ie detects a large flat surface where the regular image reports a speed sign will determine its specifically a wall mounted sign and not a post mounted sign, both require different maintained habits from each other so this is important information to know).

In a preferred embodiment LIDAR data is recorded at the same moment can be optionally used to not just get a location of an asset via correlating recorded points but also analysed to deter main traits about the asset (i.e. wall mounted vs post mounted or determining visibility such as obscured or clearly visible). TES also will analyse data about the road condition using LIDAR which point cloud data from it will be analysed and will be the main way of determining potholes and the like.

Optionally a convolutional neural network (CNN) is employed to process image data so that frames of data are compared with a series of data derived from verified images.

In a preferred embodiment the imager performs automatic image recognition of items of street furniture or other assets, by using the convolutional neural network (CNN), and either labels the item of street furniture with a code, thereby providing it with a name, or enables an operator to designate the item of street furniture with a name.

During an initial acquisition phase of each item of street furniture, an operator is optionally provided with a preselected name and an option to verify the preselected name. This enables the operator to verify quickly a preselected name; or to choose an alternative name from a predefined menu; or to define a custom name for the item of street furniture.

In some embodiments there may also be provided a means to recognise a unique identifier or code that is mounted on or marked on or transmitted from an item fitted to the item of street furniture. So that for example, the imager is able to obtain and record an image of the item of street furniture. A practical use for this is that it enables an operator to keep track and identify assets which may not have a static GPS location, such as bins which can move in measurement of miles due to swapping locations with other bins or vandalism.

Preferably a scanner or image recognition device or the optical character reader (OCR) or some other automated sensor, is operable to interpret the unique identifier or code and associate it with the particular item of street furniture, thereby enabling the generation of street furniture digital identity data and/or immediate verification of the type or nature of an item of street furniture.

The code may be alpha numeric and/or a bar code or a Q-code or a code received form a transponder or transmitter associated with the item of street furniture. Certain types of bar code may be arranged in a vertical array so that images of the bar code may be captured quickly by a moving imaging system. Certain assets may be fitted with devices that report their location, condition and other traits that are tracked in real time; an example being “smart” bins. Such assets can be interconnected to a remote asset register using wireless transceiver and data derived therefrom can be used to track assets and update them either in real time or at regular updates.

In alternative arrangement the code is transmitted as a pulsed radio frequency (RF) signal from a radio frequency identity (RFID) device associated with the item of street furniture and which RFID device may be, for example, powered by a dedicated independent power supply which may be a solar panel.

Optionally a data field is also provided to enable the status or condition of items of street furniture to be captured and recorded, either by way of an operator input terminal or by an automated system such as an intelligent image assessment means which determines a specific qualitative parameter or the type or nature of the item of street furniture.

The database currently is set up to be able to take optional values so anything associated with an asset can be stored here that a customer would want to have connected to the asset (ie condition, past repairs, running costs).

For example the data field may require a response to questions such as:

Is the item of street furniture present?

Is the item of street furniture damaged?

Does the item of street furniture require maintenance?

Alternatively an intelligent image assessment may be performed automatically by the intelligent image assessment means which determines if a lamp is working or a post is bent or if there is a break in a continuous barrier or a street sign is missing or has been damaged.

Ideally digitised image frames, their associated GPS location data and the street furniture digital identity data are stored as digital data in a database or other automatically accessible data store which may be contained in the vehicle or which is located remotely. Digital data however is preferably transmitted as a continuous data stream or in packets or frames, via wireless data network to a remote database in order to create an asset register. Subsequently the database may be accessed and used as an asset register. Access to the register permits retrieval of location data, street furniture digital identity data and data indicating status so that once accessed data may be used by and/or updated remotely by other users who may be office based.

Optionally all assets detected can be reviewed before they are committed to the asset register. This is up to the discretion of the customer and can be changed in the server settings.

Authorised users may access and update the aforesaid location data, street furniture digital identity data and data indicating status via hardwire connections or for example via 3G, 4G or 5G networks or other wireless Internet systems, via an Internet connection or via a bespoke dedicated local area network (LAN) or website which may be, for example, in an office, such as a council maintenance department or in a local government planning department or in a law enforcement department.

Preferably the aforementioned plurality of digital image frames is manipulated by an authorised user using a data processor operating under control of software. In one user configuration data may be accessed and manipulated in order to present a continuous digital data image of surroundings which data are stored with the location data of items of street furniture. Additionally, any related data, such as a unique identifier, digital identity data and any operator defined name or status criteria of items of street furniture, may also be stored for the purposes of maintenance, inspection and automated comparison with previously obtained data.

It is appreciated that the present invention is therefore able to be incorporated in or used with the mobile imaging systems as provided in the aforementioned vehicle parking enforcement system, and that data obtained therefrom is able to be scanned and checked, independently of any assessment of parking violations.

It is further appreciated that all forms of image and other data obtained may be compressed in order to speed up transmission and in order to occupy less bandwidth as well as require less storage space. Furthermore data may be encrypted in order to ensure security. Ideally the monitoring and recording system for compiling an asset register of items of street furniture includes: an imaging means or camera system that is adapted to obtain a 360° panoramic image of surroundings. In this embodiment ideally software is operative with a data processor and is configured to receive data inputs from an operator in the vehicle so as to stitch together digital images to produce a continuous non-distorted view of a scene such as a street.

An advantage of this is that the images can be made available to third parties, such as local authorities or planning departments, for inspection or planning applications. As such this set of image data may be supplied to third parties under a contract. It is appreciated that post processing may also be carried out on data for example in order to improve machine learning algorithms.

Making available such image data, under contract, may entail modifying data which may require certain types of image data to be included or removed. Software is ideally provided in order to achieve bespoke requirements, such as for example obscuring individuals’ faces, vehicle registration numbers or names of shop fronts or hoardings in order to comply with local privacy laws. Likewise data may be encrypted in order to encode the modified data for security purposes or in order to confirm to local privacy laws.

In other embodiments may have facial image recognition software incorporated within imaging systems in order to monitor or track the whereabouts of suspect individuals. Facial recognition may be performed automatically and without input from, or even the knowledge of, an operator or vehicle driver but could be performed to notify the operator of suspects and require a user review.

In some embodiments real-time data transmission and processing but could be a thing in the future if it was for some reason required.

Automatic data updates may be transmitted to a vehicle in order to suggest a route for a driver so as to incorporate as many possible points for investigation as possible. In a similar manner various databases within different departments may be polled in order to instruct the system to harvest relevant data of a particular type, for example the state and condition of bus shelter, which advertisements are being displayed on which hoardings or billboards and the number of cars parked in large carparks so as to indicate an approximately number of available parking spaces. The data harvested in this manner may be required quickly and so may be handled in a different manner to the street furniture digital identity data.

In some embodiments there is also provided a means for determining a distance of an object of street furniture from a point on the vehicle in order to pinpoint its absolute location. The means for determining a distance of an object of street furniture from a point on the vehicle ideally includes a source of radiation and a detector to detect reflected radiation from an object of interest.

Preferably the means for determining a distance of the object of street furniture from the vehicle includes: a source of pulsed radiation, a detector arranged to receive a reflected signal and a counter for determining from the source and reflected signal a ‘time of flight’ of a radiation pulse and therefore the distance of the object of street furniture from the vehicle. This can be done with the distance predictive formula currently used in the Live version of this system where is only requires 1 image (does not have to be a 360° image).

In some embodiments a recording device, which might include a voice recorder such as a digital microphone, enables the operator to provide a definition or a name by way of an oral description which is translated and included in the street furniture digital identity data by a voice recognition system.

Alternatively the voice recognition system may be used to control a camera for example to respond to voice commands of orient in a specific direction or to zoom a camera, for example where a view is obscured, so that a voice command can be used to control or to actuate an imaging device, such as a digital zoom.

A camera interface is provided for making adjustment to camera settings without requiring an external device by enabling an operator to provide oral commands, instructions and labels.

In a particularly preferred embodiment the monitoring and recording system enables the operator to associate with a digital image, a descriptor of the street furniture so that this data may be used to generate the digital identity data. The descriptor may be generated for example by way of a touch sensitive display or a stylus on screen. In some embodiments the system includes a display with a menu that operates using predictive text or short codes for items of street furniture, such as LP for lamp post or TP for telegraph pole, thereby saving time when typing the descriptor, which once verified is used to generate the street furniture digital identity data.

Optionally the monitoring and recording system includes a menu operable by the operator to provide a label for an item of street furniture.

Optionally a thing not included in this document is the ability to manually review 360° images and generate manual entries of assets using the tools provided.

So as to speed up and simplify operation of the system, a menu or a selection of options may be presented on a touch sensitive display thereby enabling an operator for example to assess and measure a distance of an object of street furniture from the vehicle in real time. This information may be used by the operator in order to label the precise whereabouts of a specific item, for example for subsequent investigation or maintenance.

In a real-time identification scenario (future when processing speeds are extremely quick) the person could pick a sub type for an asset the computer shows on screen by a simple touch screen with suggested sub types

The monitoring and recording system may be incorporated in an existing road inspection system or parking monitoring system or mapping system or it may be deployed in an autonomous vehicle, such as a driverless car or a drone or police helicopter.

The system is a complete asset recording and monitoring system on its own but can be integrated to work with third party system for data sharing and processing requests due to optional customised features.

A particularly preferred embodiment of the monitoring and recording system includes: a comparator which compares images or portions of an image; and voice recognition and response system that responds to a voice command; and a display which enables an operator to manipulate a scene in order to view an image showing amongst other things items of street furniture. In a particularly preferred embodiment a scanner scans a scene and obtains stored data and presents image data in a continuous format with the aforementioned digital identity data and operator defined name and status of items of street furniture. Once images are scanned and combined with data and stored on the database they may be retrieved for example for maintenance planning or in order to determine if an object is missing, damaged or faulty. In order for a CNN to be able to locate an object it must be trained with images of the object in question so as to build a weighting function which act as an area of object association on a graph. The rule of training is that the more images obtained the greater is the accuracy of identification.

Ideally the system receives a frame of data from a camera or imager and a frame of data is fed into the CNN via a CNN input. Processing of the data then takes place in the CNN after which a dictionary of details is output from the CNN. The dictionary of details contains the class of object to which the imaged object belongs and a mask of the object in the image. For example the image processing system determines if the imaged object is a lamppost or telephone box. The probability of the object being in one class of objects is determined by a number of factors including the position and size of a boundary box that surrounds the object. Using the finding from the LIDAR data we can then also add more traits the asset has into the final classification if needed.

The CNN takes normal images and processes those finding shapes and common patterns. The CNN is trained on these images previously and generates a "weight" file which is used to adjust its internal calculations in ways that allow it to say which pixel belongs to a class and to which class. The mask RCNN uses these pixel notes to generate a "Mask" for segmentation so it is possible to see where one asset starts and another one finishes. Image data may be manipulated (encrypted or compressed) or processed (scanned) or digitised locally in the vehicle or remotely at a data centre.

According to another aspect of the invention there is provided method of compiling an asset register of items of street furniture comprising the steps of: operating a vehicle mounted camera in order to derive a plurality of image frames; generating a continuous digital image of surroundings; associating the assets in the image with a location derived from a global position system (GPS); providing location data which is associated with each image frame; deriving from the images identity of items of street furniture and associating with each item of street furniture a unique identifier; and/or the option of transmitting image frames, GPS data and the unique identifier as digital data to a remote database.

It is understood that aspects of the system may be incorporated into the method as appropriate.

Preferred examples of the invention will now be described, by way of example only, and with reference to the Figures in which:

Brief Description of the Figures

Figure 1 shows a monitoring and recording system according to the first aspect of the invention in use on a first earlier date;

Figure 2 shows an example of a typical screen output which is a schedule of recorded items of street furniture, with their unique identity codes and their GPS locations and their locations corresponding to the instant of Figure 1 ;

Figure 3 shows the monitoring and recording system of Figure 1 in use at the same location on a second later date;

Figure 4 shows typical screen output which is a schedule of recorded items of interest and their locations corresponding to the instant of Figure 3;

Figure 5 shows a print out of a verification report corresponding to the typical screen output which is a schedule of recorded items of interest and their locations;

Figure 6 shows an overview of one example of a monitoring and recording system with an asset register and a mobile vehicle mounted imaging system; and

Figure 7 shows a monitoring and recording system in use and illustrates diagrammatically how the LIDAR operates to identify a plurality of examples of street furniture.

Detailed Description of Preferred Embodiments of the Invention

Referring to the Figures generally, and in particular to Figure 6, there is shown an example of a monitoring and recording system 10 according to the first aspect of the present invention which in use obtains image data, processes the image data, and transmits it in a digitised format via a data network 66 to a remote location where it is compiled to produce an asset register of items of street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28, and 29. The asset register 44 is stored on a database 50 and is accessible remotely by one or more users (not shown).

Asset data is stored in a database which is hereafter referred to the asset register. The database may be queried to filter results or to obtain assets returned to the user. Systems may use relational based database structures. However current and future plans use Mongo DB NoSQL non-relational document structure due to their ability to be scaled up easily and to adapt to changing data input compared to the fixed schemes that SQL enforces.

The system 10 comprises: a vehicle 12 mounted 360° camera system 30 that includes a plurality of cameras and which obtains frames of image data; two, three or four additional pulsed radiation transponders 31 , 32, 33 and 34 which acquire reflected radiation from imaged objects; and a data processor 38 which calculates distances of objects from the vehicle and sends this information to an on-board data store 29.

Image data is processed by the data processor 38 and transmitted in real time with other data files to a receiver 43 from where data is relayed, via a data network 66, to a database 50. Alternatively, data may be transmitted as an entire file when the vehicle 12 is able to be connected to the database 50 or a related system via a hardwire cable (not shown). The image data and files are used to add/update the asset register 44 or to be reviewed.

As described in detail below, and with reference to Figures 1 and 6, images of items of street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28 and 29 are initially obtained during a first acquisition phase, for example on the first occasion when the vehicle 12 drives along a road and when an operator first identifies and labels each item of street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28 and 29.

During an initial acquisition phase, the operator is optionally provided with a preselected name of each item of street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28, and 29 as it is imaged and labels each of the items with a name. A menu, such as that depicted in Table 1 below, is presented on a touch sensitive display 36 of a portable device, laptop or palm pilot 14. The menu provides the operator with options for selection and naming each item of street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28, and 29 detected by the system or an option to verify a preselected or preassigned name or choice of names or short hand codes as previously described. This may be done using a touch sensitive display or with a mouse and microphone.

For example, the menu might offer the choices for the items of street 17, 19, 21 , 22, 23, 25, 26, 27, 28, and 29 which may be presented as a table or menu on the screen 16 as depicted in Table 1 below.

Table 1 is able to store other data like asset image location in its respective database, for using an“assets_<customer name>/2019-04-04/#A1263537. png”

Table 1

Subsequently the name of the street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28, and 29 and their unique identifier codes are compiled and transmitted as digital data to a data network 66 which once relayed to a database 50 may be accessed by users from remote terminals accessing databases in a control room or secure data centre.

Further choices may be generated and provided by using a neural network and artificial intelligence systems which in combination with computer imaging, pattern recognition and machine learning systems are adapted to interpret and label automatically images obtained by the cameras of the camera system 30 so that an operator merely has to affirm the identity or name of an item.

A unique identifier or code 24 is optionally also marked on an adhesive label or integrated in some other way with some or all items of street furniture so that for example pulsed infra-red (IR) radiation from transponders 31 , 32, 33 and 34 are able to record the unique identifier or code 24 and the distance of an item of street furniture and image recognition device or optical character reader (OCR) or some other automated sensor, is operable to interpret the unique identifier and associate it with the particular item of street furniture and thereby generate the street furniture digital identity data which unambiguously identifies each item of street furniture rather like a finger print.

Use of the invention ensures unambiguous identification of each item of street furniture 17, 19, 21 , 22, 23, 25, 26, 27, 28, and 29 and its precise location. In use image data is provided to an operator who is enabled to verify quickly the name of each item of street furniture from a preselected list of names or types of street furniture as well as its state of repair.

The image recognition software advantageously limits the number of choices of names for an item of street furniture from a longer menu. Alternatively, names of street furniture are selected from a predefined menu or in the event that image recognition is unable to identify a particular item of street furniture a custom name for the item of street furniture may be input by the operator.

Frames of captured image data are digitised, time and date stamped and labelled with a GPS marker. Digitised images together with other data are then transmitted by transmitter 42 to a database via a base station 56 and network 66 or directly to a remote receiver 43 where they are stored and uploaded to a database 50. One stored on the database data relating to digital images of surroundings may be obtained and manipulated. Thus, a database 50 may be interrogated and viewable by SQL or similar imaging processing software that obtains stored data and presents the image data in a continuous format on a monitor 46. Referring to Figures 1 and 3 a global position system (GPS) 40 provides location data of the absolute whereabouts of the pulsed radiation transponders 31 , 32, 33 and 34 so that location data is associated with each digital image frame.

The pulsed radiation transponders 31 , 32, 33 and 34 thereby derive distance of items of street furniture at locations along a route taken by the vehicle 12. The GPS time, date and place stamps each image. Image data is stored on an on-board data store 29 and linked than item of street furniture from a unique identifier; and a transmitter transmits image frames, GPS data and the unique identifier as digital data to a remote database which stores the location, GPS data, and unique identifier and an operator defined name for the item of street furniture.

Outputs from digital cameras 30a, 30b, 30c and 30d provide a continuous digital image data of surroundings. Global position system (GPS) 40 provides location data which is associated with each image frame and transmitted therewith. Thus the data showing the type and nature and location of street furniture, when combined with the unique identifier 24, can be obtained, digitised and stored continually (day and night) and prided as incremental updates to a database.

Image recognition software, running on server 48, is operable to derive an identity of each item of street furniture from its unique identifier 24. If no unique identifier 24 is present or detected, a suggestion of the nature, type, name, condition or type of street furniture is provided as a menu for the operator to select or to verify Al generated estimations, as described above. In the event that no identity or name can be provided, a digital flag is associated with a file for investigation at a later time. This may involve sounding an alert to the vehicle driver to halt at a particular location and investigate an item for naming or inspecting.

The transmitter 42 transmits image frames, GPS data and the unique identifier 24 as digital data to a remote database 50 which stores the GPS location data and the street furniture digital identity data.

Referring to Figure 1 there is shown a diagrammatic overview of one embodiment of a monitoring and recording system 10 which was obtained on a first earlier date. A unique identifier 24 is associated with the lamp post 28 as the operator is provided with a short hand code‘LP’ which may be presented as 11 1 , as shown in the Table. However, the system 10 recognises that this particular lamp post 28 is at a unique position from its GPS location coordinates and so assigns the unique identifier #A1263537 as a CSV. In use an operator defined name may also be input by an operator, for example using a touch sensitive display or panel 36 on a portable device, laptop or palm pilot 14 which provides the operator with selection options in order to verify a preselected name and thereby generate the unique identifier.

Referring to Figure 2 there is shown a print out of a screen shot of a schedule of recorded items of interest (assets) and their locations. The list has been derived from images obtained from the cameras 30a, 30b 30c and 30d and from pulsed radiation transponders 31 to 34, and occasionally augmented by an operator command. This data is transmitted and stored on the database 50 from where it can be retrieved by a user (not shown) and printed.

By way of example some time after acquisition of the data shown in Figure 1 , the image of Figure 3 illustrates a diagrammatic overview of the scene shown in Figure 1 and obtained on a second later date.

In the scene shown in the example depicted in Figures 1 and 3, street lamp 28 has an RFID 18 transmitter mounted on its upper surface which is powered by a solar array (not shown). The RFID tag 18 transmits a pulsed coded signal which includes the GPS location and identity and status of each item of street furniture. The coded signal is detected and decoded and associated with the street lamp and a record of its status is associated with its digital identity data thereby providing a record of its location and an indication of its status/condition.

Figure 4 shows typical screen output which is a schedule of imaged and recorded items of and their locations. In particular the absence of pothole 20 which was present in a previously acquired image (Figure 1 ). On comparison of the two images in Figures 1 and 3 it can be deduced that the pothole has been repaired. This may be important from a perspective of providing an audit of road works/road repairs

Figure 5 shows a print out of a verification report corresponding to the typical screen output which is a schedule of recorded items of interest and their locations. There is shown a table or print out which has been obtained by comparing either two images or a catalogue of items represented as digital data or by both techniques in order to arrive at a discrepancy or difference.

The image in Figure 1 is of a scene that was obtained on 20 November 2017 as can be seen from the data stamp at the top left hand corner of the image. The image in Figure 3 is shown as having been obtained on 2 February 2018, again from the date stamp at the top left hand corner of the image.

Data from digital scanner 48b and image recognition system 60 are compared in a comparator 49 which compares scanned image data and data provided by the image recognition system 60, from images obtained at different times (Figures 1 and 3) in order to assess the status of an asset and determine for example if the asset (item of street furniture) is missing or damaged or faulty. The example shown in Figures 1 and 3 shows a pothole 20 (in the bottom left hand corner) as being present on 2 November 2017. In the view shown in Figure 3 the pothole is shown as having been repaired and therefore a completion report may be raised and transmitted to a local authority or customer or to an auditing authority.

Figure 6 shows an overview of one example of a monitoring and recording system 10 which is optionally used in surveillance. Remote updating of a digitised street map, stored as an asset register 44 on the database 50, may be made via input terminals (not shown) for example in local planning departments or council maintenance offices in order to provide an update obtained from other sources enables an image obtained from data from the digital scanner 48b and image recognition system 60 are compared in a comparator 49 which compares scanned image data and data provided by the image recognition system 60, in order to indicate a difference, updated event or feature for confirming, for example whether a piece of maintenance or repair work has been carried out. Optionally an automated procedure may be provided for this purpose with an option of a human audit or‘sign off being provided.

Figure 7 shows an alternative embodiment of a system that operates using a LIDAR system 35 to detect and measure distances to solid objects. LIDAR is a surveying technique in which is used to measure distances from a source to a target object. This carried out typically by illuminating the target with a pulsed laser beam and measuring reflected pulses with a suitable sensor. Differences in return times of pulses and data derived from interference of waves is used to determine the distance to the target. LIDAR may thereby be used to create a digital three-dimensional (3-D) representation of the target and/or an environment monitored by the system.

By employing a LIDAR system 35 with a GPS location system, it is now possible to identify an object (asset) and its location and record this data on a digital map. Previously this was a multi stage operation involving a surveyor (individual or an automated system) locating an asset, determining what type of asset it is, optionally determining the asset’s condition, and then recording this data in a database. Needless to say, this entire process was labour intensive and therefore expensive. There was also a risk that errors would be introduced into the data that was captured and recorded.

Ideally image and location data which is derived using the system is stored on a database or optionally on a bespoke database with machine learning capability, so that an existing object can be associated or correlated with image and location data derived from the system. This data enables a real time graphics feature to be provided to a user so that the user is able to“snap to map”, for example by clicking a mouse, and so identify (or verify) an existing object and confirm its exact location with respect to a closest GPS location in order to verify data on the database or initiate a query for an operator to verify later.

An advantage with the invention is that because it operates automatically multiple detections may be performed simultaneously using two or more, and preferably six, imaging systems which operate in parallel. Preferably software is operative to knit together images and/or location data, thereby providing a virtual image environment, for example for a remote inspection team and/or to compress data in order to enhance the data for storage or transmission.

The system may use artificial intelligence, which may include image recognition software, so that certain objects, such as potholes, generate an action warning which in turn may result in a report being generated with a series of check lists or activities, as well as an optional automatic transmission of a short message service (SMS) alert to a maintenance or inspection team; for example, when multiple detections of such certain objects have occurred at the same or at nearby GPS locations or when particular damage or faults are identified.

As in the previous embodiment, the LIDAR system associates a unique asset ID with all items detected, for example, so that a register of assets can be compiled.

Ideally image recognition software enables non-detected items, which have been marked as financial assets, to be flagged and optionally an alert to be issued so that personnel can be despatched to investigate and survey the absence of the marked non-identified item. Alternatively, or additionally, the system may flag detected items which have not been marked as financial assets and may optionally issue an alert so that personnel can be dispatched to investigate survey and authenticate the previously unknown or un-registered asset. As data includes a GPS location, it is also envisaged that software will be employed to generate a preferred route along which investigating personnel may walk, or drive, so as to visits as many locations in a shift as possible.

As images are acquired using a 360° camera zooming into and around objects is achievable in real time or later so as to enable detailed inspection of an asset. This inspection is important for repair and maintenance purposes.

Images obtained using the system are ideally watermarked with data and timestamps which include at least the following: date and time of acquisition, local ambient weather and light conditions and GPS location. Watermarking may be imprinted on image as text or another distinguishable mark and/or may be imprinted as encrypted data, such as in bar code or QR-code format.

By using both LiDAR and high definition (HD) 360° imaging the user has the ability to serialise all assets that possess a monetary value, such as bicycle racks, street railings; as well as to provide a community service by detecting potholes, litter bins placed on sidewalks which may present as a hazard; or by detecting items of street furniture requiring maintenance and repair. This is achieved in a short time period at relatively low cost.

Most companies that provide asset registers tend to manually count street furniture over a period of months and this tends to cost to be in the hundreds of thousands of pounds. This also means the register will never be accurate as these counts only take place every 10 or so years.

In addition to detecting items of street furniture, the system may also detect, image and locate stationary or mobile other vehicles such as cars 70 or bicycles 72.

It will be appreciated that variation may be made to the above-described embodiments without departing from the scope of the invention as defined by the claims.

For example, the system may monitor fly tipping, unauthorised skips, dumped items of furniture, or abandoned cars which when imaged attract an automatic flag which can be relayed to an inspection team for subsequent inspection, such as during quieter traffic times or at weekends, and/or when time permits for the removal of such items.

Other variations include obtaining images from one or more unmanned aerial vehicles (UAVs) or drones in order to image large car parks and/or places which may have temporarily restricted access or are inaccessible by vehicles, such as a construction site, crowded areas, or a pedestrian zone which might impede traffic movements during busy periods.

List

10 system

12 vehicle

14 on board personal digital assistant

16 on-board microphone

17 tree

18 RFID tag

19 traffic light

20 pothole

21 bus stop

22 manhole cover

23 bin

24 unique identifier

25 bench

26 guard rail or barrier

27 Sign

28 lamp post

29 traffic cones

30 a vehicle mounted 360° camera

31 first pulsed radiation transponder

32 second pulsed radiation transponder

33 third pulsed radiation transponder

34 fourth pulsed radiation transponder

35 LiDAR system touch sensitive display data processor

global position system (GPS) radio frequency (RF) transmitter radio frequency (RF) receiver asset register

monitor

a computer

b digital scanner

image comparator or data comparator remote database

base station

image recognition system

data network

car

cyclist