Title

ELECTRONIC MUSIC EQUIPMENT HAVING A SIMPLIFIED COMPOSITIONAL INTERFACE k k: k: k: k: k:

Technical field

The present invention belongs to the sector of electronic equipment which can be programmed to emit sounds, and in particular the invention belongs to the sector of equipment which, by means of a simplified user interface allows to compose a music piece which is subsequently automatically memorized and reproduced by a piece of equipment which processes and amplifies the signal and transmits the latter to one or more loudspeakers.

The equipment according to the present invention can, preferably but not exclusively, be used for didactic purposes in order to teach children the music fundaments such as rhythm, melody and harmony.

Present status of the art

Solutions for automatically transforming a musical text in a music file and/or in a melody which can be automatically played by an electronic equipment are known; in particular some known solutions use a digital camera for acquiring the music text image and afterwards use instruments of informatic type which automatically transform the image in a music file which subsequently is played by way of amplifiers, loudspeakers and similar devices of a known type.

All these systems require music be written using the traditional symbology, typically by using staffs for writing the graphical signs which represent the notes and all the other symbols traditionally used for writing a music piece. Hence, when known systems are used, in order to reproduce a sound it is necessary to have a basic knowledge of music as well as the symbology for writing music.

Objects and summary of the invention

A need is therefore felt to have an equipment capable of easily composing a music piece without knowing musical writing and symbols, in particular a need is felt to have a simply equipment capable of teaching novices, in particular children, the meaning of basic concepts of music composition such as, height of notes, their duration, and harmony.

An object of the equipment according to the present invention is to provide children the elements with which they can make each time a different object which in its turn represents sounds, alternate to a staff.

Another object of the equipment according to the present invention is that of converting a visual image into a melody, by analyzing a photogram of a compositional plate on which modular elements are arranged.

In particular, the composition of a music piece takes places by horizontally distributing on an orthogonal Cartesian plate standardized modular elements, each of which represents a note. The length of each modular element is a multiple of a predetermined unitary module and represents the duration of the note; the height of the later is instead determined by the ordinate in which the modular element is arranged in the Cartesian plane; the start and end moments of each note are defined by the abscissas where the modular elements start and end.

Each modular element is arranged horizontally and identifies an ordinate only, i.e. a height only.

The orthogonal Cartesian plate is extended vertically up to comprise a plurality of octaves and/or fractions of octaves. The lengths of the modular elements are a multiple of the basic module and vary in the range at least from one to four; traditionally the base element, made up by only a module, has a duration of a fourth, or submultiples, whereas the element made up by four modules has a equal duration to a beat of four fourths, o submultiples.

It is worth pointing out that the different modular elements can be considered modules of a fourth, two fourths and so on, but, if musical time of the piece is changed, the basic module might become a submultiple of a fourth, for instance an octave or a sixteenth, in this case the other modular elements would vary consequently.

It is also possible to use modular elements having a length greater than four modules, for instance an equal length to five or six modules or even more.

In order to make the subsequent optical recognition easier it is advisable that all elements having the same duration be of the same color, different from the color of the other elements having a different duration; also, all colors of the same modular elements are different from the color of the orthogonal Cartesian plate on which the modular elements will be placed to compose the music.

So, two horizontally adjacent modular elements differ by only one element having the same overall length by the fact that the former represent two notes played in succession, whereas the latter represents one single note, the duration of which is the sum of the durations of the two notes represented by the two elements approached to each other.

A computer visual recognition system acquires the structure of the generated composition and, by way of an object optical recognition system, it associates a sound and a duration with each shape and position wherein a modular element is present, thus reproducing the musical sequence created. According to a comfortable and practical embodiment the orthogonal Cartesian plate is placed internally to a box like body the upper face of which can be opened like a lid hinged on one of the upper edges of the box-like body itself. After opening the lid, the modular elements can be placed on the Cartesian plane, the upper face of the box-like body comprises a digital camera, electronically connected to a computer vision system.

The box-like body is configured in such a way that if its upper face is opened, once having placed the modular elements on the orthogonal Cartesian plate, the digital camera focuses the orthogonal Cartesian plate and subsequently, by pressing a start push-button, it acquires the image; afterwards an image recognition software isolates and recognizes the different modular elements, and associates with the color, the length, and the position of each element the corresponding note, as well as the duration and its beginning instant.

It is worth pointing out that recognition of the duration of notes might take place not only in accordance with the color, but also in accordance with any other characteristic identifying the length of the modular elements, as long as it is discernible by the recognition software.

In a preferred embodiment of the invention the extrados of the surface of the orthogonal Cartesian plate presents engagement means suitable for reversibly connecting to relevant engagement means present in the lower portion of the modular elements, in such a way that the latter maintain their position after being placed on the orthogonal Cartesian plate.

The engagement means define rows and columns, typically each row corresponds to the height of a note and each column corresponds to a duration of a fourth of beat.

For this reason, preferably, the columns of the orthogonal Cartesian plate are in a number that is a multiple of four. On the box-like body there is also a push-button to stop the above-described process; in a particularly accurate embodiment of the invention, on the box-like body there are also indicator lights which provide additional information, for instance an indicator light is steady if the data processor is ready to start the process, whereas it is flashing if the data processor is starting up or is in the shutdown step; upon reaching the latter status the indicator light turns off.

Likewise, another indicator light can show the status of processing.

The image acquired by the digital camera is transmitted to a data processor, which processes the image in order to recognize the type and position of each modular element, thus having the information necessary to arrange a set of notes which, by way of a subsequent processing, the data processor transforms into an audio file that is sent to the loudspeakers through an amplifier.

According to a particularly effective operating procedure, upon starting gaming and after positioning the different modular elements on the plate to form a visual melody, the digital camera acquires an image of the plate.

To keep an optimum accessibility to the plate that forms the compositional area, in order not to interfere with the environmental lighting and to keep a good general ergonomics of the equipment, the digital camera is advantageously housed inside the lower surface of the lid which, once opened, puts it over the plate to be shot. According to a particularly advantageous solution, the digital camera is placed in correspondence with the edge of the lid opposed to the hinged side and is inclined approximately 20° with respect to the assessment area, in order to promote the total acquisition of the whole plate and to limit, already at this early step, the light gradients which perturb the image acquisition. In order to reduce the equipment price, it is possible to use a low-quality digital camera and because of its non central position the resulting photogram often has a high perspective distortion which requires to make an operation of real time preprocessing of the acquired image, in order to correct disturbances and restore the correct proportions, also taking advantage of some graphical elements which act as references such as, for instance, the perimeter of reference of the image.

The equipment simultaneously performs the binarization operations with dynamic threshold, aiming at zeroing the light gradient, which is very detrimental during the recognition step of the colored modular elements, the management of which is not beforehand determinable because it depends on the light conditions wherein the equipment is allocated; as a matter of fact, depending on how the modular elements are lit and on the material they are made of, they reflect and diffuse the light in a different way, so they create overexposed white zones which are difficult to manage afterwards.

Taking advantage of the fact that distance and inclination of the digital camera are known and that the orthogonal Cartesian plate has both its position and dimension fixed, the algorithm of the data processor creates a virtual grid, superimposed to the real plate, which precisely identifies every single engagement means of the modular elements with respect to the plate.

Then a second virtual grid is created starting from the first one, by identifying the reciprocal distances amongst every engagement means and creating around the latter a number of polygons having a fixed dimension, which corresponds to the plan-view dimensions of the modular elements which can be placed therein. Account being taken of the different probable lighting conditions and of the removal of the light gradient, a number of preliminary statistical analyses have been performed on the colors of the selected modular elements (for instance: white, yellow, red, orange) and on the color of the plate itself (for instance: blue), identifying a color of the RGB family representative for each of them and resulting from the average of the colors of the modular elements present on the plate and of the color of the plate itself, in the different most common lighting possibilities wherein the equipment will presumably be used.

At the beginning of the process, hence after acquiring the image and after removing distortions, the algorithm calculates once again the second virtual grid on the image itself and begins a scanning algorithm and object detection row after row.

As a not-binding example we state that the different columns are analyzed in succession, from left to right, from the lowest row towards the highest one in the matrix.

For every element of the grid the average of the triad of colors of the pixels of the image is calculated and said average is then compared with the patterns of sample colors, by way of Euclidean distance in the RGB space. At every element of the grid the color of the pattern which corresponds to the lowest Euclidean distance is subsequently assigned. In addition, to speed up the recognition step and reduce its expensiveness the information of the length of every modular element is used, deriving from the assessed color. As a matter of fact, the image processing resume from the first element of the grid following that which the algorithm supposes to be the last of the considered modular element, so as to further optimize the calculation time. Lastly, the software, once the timing scheme of the notes is rebuilt, simultaneously with the recognition of the position along the vertical axis and hence the associated note, and the duration of the modular element, transforms such sequence in a MIDI (Musical Instrument Digital Interface) file which is ultimately reproduced by way of a computer, an amplifier, and one or more loudspeakers, placed internally to the equipment.

Besides the above-described method there are other possible approaches for processing in a particularly effective way the image shot by the digital camera, with particular reference to the different possible lighting conditions which might make it difficult to identify the color of the modular elements. For example, an approach that preliminarily includes an automatic learning by the system ("machine learning") based on the concepts of neural networks is possible; in this event, the method is anticipated by the following steps which will be performed only once by the technicians that set up the equipment and, therefore, said steps do not have to be performed by the normal user.

- implementing the data bank to be used for the training of the neural network with supervised learning: this step is performed only once and it is necessary to train the neural network to classify the modular elements on the basis of their color, minimizing the perturbation variables introduced by the different lighting conditions. The data bank is implemented starting from the acquisition of images of monochromatic plates of a known color, under different lighting conditions. The higher the number of samples used to implement the data bank, the higher the neural network classification. For each predefined color its respective data bank should be created and memorized. The samples of the data bank are formed of sequences of colorimetric triads, which correspond to RGB colors of predefined portions of the plate. Each acquisition provides a number of samples equal to the product of the possible abscises for the possible ordinates of the plate; in a plate as that shown in the attached figures, having 16 abscises and 16 ordinates, each acquisition provides 256 sample colorimetric triads.

- training the neural network: when the data bank has reached a sufficient number of acquisitions, the processing step necessary for training the neural network is performed. In this step the composition in terms of layers and perceptrons of the network shall be decided, identifying, by means of tests and attempts, the best configuration. Therefore, part of the samples of the data bank will be used in the supervised training step and part in the validation step, by monitoring the error value in order to prevent the so-called overfitting phenomena.

According to a particularly practical and functional embodiment, the orthogonal Cartesian plate can be removed from the equipment according to the present invention and when it is separated from the remaining part of the equipment, the colored modular elements can be arranged on it. In this way, more children can individually compose one music piece of his/her own in an independent manner arranging the modular elements on a respective orthogonal Cartesian plate; subsequently the orthogonal Cartesian plates are progressively inserted into the equipment which, for each orthogonal Cartesian plate, creates its respective musical files and executes them.

Brief description of the drawings

Figure 1 shows an axonometric view of the equipment according to the present invention, without the orthogonal Cartesian plate, with the box-like body (8) in the open position, and the lid (10) in a raised position; the digital camera (3) is visible on the top of the lid, the socket (87) for recharging the battery is visible on the frontal wall of the box-like body. The cross (11) on which the orthogonal Cartesian plate rests is also visible.

Figure 2 shows the orthogonal Cartesian plate (1) along with a sample of the four different types of modular elements, the length of which is of one module (21), two modules (22), three modules (23), and four modules (24) respectively. Figure 3 shows an axonometric view of the equipment, the lid being in the closed position.

Figure 4 shows a side view of the equipment; the lid (10) of the box-like body is visible in the open position, on its top the panel which protect the digital camera (3) is also visible. On the side face of the box-like body are identified the control (83) for adjusting the volume during the execution of the music piece, the second indicator light (85), and the switch (86) for turning the equipment on and off.

Figure 5 shows a top view of the equipment according to the present invention, without the cross (11) which supports the orthogonal Cartesian plate (1); the image also shows the data processor (4), the audio amplifier (5), two loudspeakers (61, 62), the power supply device (7), the box-like body

(8).

Figure 6 shows a top view of the equipment; this image also shows the cross (11) which supports the orthogonal Cartesian plate (1); the image also identifies the control (81) for starting the acquisition of the image and for executing the music piece, the control (82) to switch the equipment operation off, and the first indicator light (84); in the image the reference perimeter (9) is also identified.

Figure 7 shows the image A acquired by the digital camera (3), with the distortion due to parallax; the figure also shows one of the engagement means (12) of the modular elements.

Figure 8 shows the image B after the operation of straightening performed by the data processor (4); the figure also shows the orthogonal Cartesian plate (1) on which six modular elements (21) having a length of one module and eight modular elements (24) having a length of four modules are arranged; the figure also identifies the control (81) for starting the acquisition of the image and for executing the music piece, the control (82) to switch the equipment operation off, and the first indicator light (84); the figure also identifies the reference perimeter (9) of the image. Figure 9 shows the image C, put inside a reference perimeter (9) and subdivided in boxes.

Figure 10 shows the image D and schematizes the results of the executed processing, identifying each note and its starting moment.

Detailed description of embodiments of the invention

The following detailed description is given for explanatory non-limitative purposes only and makes reference to the attached drawings, and highlights further characteristics and advantages deriving therefrom and which are an integral part of the subject invention.

The electronic music equipment according to the present patent application comprises a box-like body (8) which an orthogonal Cartesian plate (1) is associated to, where the extrados includes engagement means (12), advantageously arranged according to horizontal rows and vertical columns, suitable for reversibly connecting to corresponding engagement means present in the lower portion of a plurality of colored modular elements (2).

The length of each of said colored modular elements (2) varies in the range from one module to at least four modules, whereas the width is always constant; all the modular elements having the same length (21, 22, 23, 24) are of the same color, and the color of the modular elements of every length is different from the color of the other modular elements having a different length.

The color of the orthogonal Cartesian plate (1) is different from the color of each of said modular elements (2).

To better understand the importance and advantages deriving from the fact that all the modular elements having the same length (21, 22, 23, 24) are of the same color and that the color of the modular elements of each length is different from the color of the other modular elements having a different length, it is worth pointing out that otherwise it could not be possible to distinguish multiple adjacent modular elements from one modular element featuring a length that is equal to the sum of the lengths of said multiple adjacent modular elements.

In that case, the device, instead of correctly reading plural consecutive short-duration notes, would read one note of a duration equal to the sum of the plural short durations.

A digital camera (3) is connected to the box-like body (8); this camera focuses the orthogonal Cartesian plate (1) along with all the colored modular elements which are reversibly connected onto its extrados; the digital camera (3) produces an image, in the form of a digital file, and transmits it to a data processor (4) accommodated in the box-like body (8). The data processor (4) processes the image received from the digital camera (3) and retrieves the number of modules of each modular element, besides its position on the orthogonal Cartesian plate (1), and finally provides a music file which transmits to an audio signal amplifier (5) which, in its turn, transmits it to at least one loudspeaker (61, 62).

The equipment according to the present invention also comprises a power supply device (7); the latter can be a transformer directly connected to the mains or, advantageously, it can be a battery accommodated internally to the box-like body (1) and rechargeable through a socket (87), for instance an USB port.

In a particularly complete embodiment of the invention, the amplifier (5) is interfaced with a user interface comprising:

- a control (81) suitable for starting both the acquisition of the image and the execution of a thus obtained music piece;

- a control (82) to switch the equipment operation off;

- a control (83) for adjusting the volume during the execution of the music piece;

- a first indicator light (84) which, if steady, indicates that the equipment is ready to be used, whereas if flashing, it indicates that an image acquisition is in progress;

- a second indicator light (85) which indicates if the data processor is operating, switched off, or in a transient step. In the embodiment shown in the figures, the modular elements (2) are shaped as parallelepiped blocks, wherein the lower portion of each module defines a female cavity in which a corresponding cylindrical protrusion that rises from the extrados of the orthogonal Cartesian plate (1) engages. The image recognition step is particularly delicate and good results have been obtained by applying the method described here below with reference to the attached drawings.

- Composing a music piece by arranging the modular elements (21, 22, 23, 24) on the orthogonal Cartesian plate (1).

- If the orthogonal Cartesian plate (1) is not already internally to the box-like body (8), it should be inserted, in the relevant zone, internally to the reference perimeter (9).

- If the indicator light (84) indicates that the equipment is ready to be used, starting the image acquisition by the digital camera (3) by pressing the start control (81); the indicator light (84) will flash for all the duration of the processing .

- The data processor (4) acquires the image file A obtained by the digital camera (3).

- The data processor (4) straightens the image A thus reducing the effects of parallax and obtaining the image B.

- The data processor (4) trims the image B and puts it inside the reference perimeter (9), thus obtaining the image C, which subdivides in boxes.

- The data processor (4) performs the analysis of the colors of the plate (1), subdivided in boxes, and afterwards saves the thus obtained image D.

- For every assessed box the data processor (4) performs the analysis by processing the average RGB value of the box. For every calculated average value, the distance with respect to the four reference colors is calculated; if the obtained distance is smaller than the threshold value, then the box is associated with that color. The duration of the note is defined in accordance with the assessed color, its height in accordance with its ordinate (axis Y), and the instant of its execution in accordance with the abscissa (axis X); then the note is inserted in the MIDI file of the composition.

- The data processor (4) memorizes the image D along with its relative MIDI file in an image and audio archive of the created compositions.

- At the end of the processing of all the grid, the data processor (4) starts the execution of the thus generated MIDI file, in loop, up to the activation of the control (82) which switches the equipment operation off.