1 Background of the invention

1.1 Field of the invention

The present invention relates to augmented stringed musical instruments, i.e., conventional acoustic stringed instruments enhanced with electrical sensors and actuators.

1.2 Background art

The internet of things revolution has impacted a variety of fields and industries, including acoustic stringed instruments. As a result new augmented instruments or prototypes were introduced. Prior art shows several examples of augmented stringed musical instruments and different applications (US 20140224099 Al; US20120007884; US2002/0005108A1; PCT/GB2000/000769; US 6,320,113 Bl; US 8,389,835 B2) .

The problem addressed relates to the need of players of acoustic stringed instruments to enhance the range of sound, timbre and vibration of their instrument in order for instance to replicate the resonance or other characteristics of another instrument or to recreate special sound or vibration effects, or to modify such effects during the act of playing. The existing technologies available to acoustic stringed instruments allow to create special effects only by acting on the physical feature of the instrument, i.e. the size, or the shape or the thickness of the sounding board, the material of the different instrument's parts, or the size of the strings, which is obviously an operation that can be done exclusively before the act of playing and which requires often a recrafting of the instrument.

2 Summary of the invention

The present invention provides an innovative solution to the described need. The invention describes a method and a process to synthesize a desired vibration or timbre or sound of another instrument (Target Instrument) in real time. The solution will make possible, for instance, to make vibrate a stringed instrument (Controlled Instrument) similarly to the timbre of another stringed instrument (Target Instrument), this last one having different features and a different timbre than the Controlled Instrument. The capability to control and modify such effects will be accomplished in real-time while the musician is actually playing the instrument.

The Controlled Instrument and the Target Instrument are stringed musical instruments that include a soundboard and/or a sound box, and a bridge. A measurement apparatus of the instruments vibration is placed under the bridge of the Controlled Instrument, and two actuators are attached respectively to the main resonating surface and a secondary, mechanically uncoupled tonal chamber. The signals driving the actuators are determined by a Control Algorithm. While the presence of a secondary actuator does not improve the generality of the system, its contribution may become fundamental in practice when using one of the empiric estimation techniques hereby proposed. The Control Algorithm imitates the Target Instrument by feeding the actuators with a processed version of the Controlled Instrument's string and soundboard vibration detected by the measurement apparatus while the instrument is being played. The Cloning Procedure gives the parameters of the Control Algorithm when such an algorithm has to control the Controlled Instrument so it sounds similar to the Target Instrument. However, the Control Algorithm is not restricted to only impose the reproduction of a Target Instrument. The Control Algorithm can make the Control Instrument sound in other desired manners by choosing the Control Algorithm parameters.

The patent US2002/0005108A1 describes several sensors and actuators embedded in a musical instrument, which are capable to control the sound emission, among other features, via a DSP (Digital Signal Processing) module. The DSP unit implements the sound controller. The Control Algorithm of the present invention needs the sensor/DSP/actuator architecture described in the patent US2002/0005108A1. However, such a patent does not specify neither the Cloning Procedure, nor the Control Algorithm, which are instead the inventive steps of the present document. The patent PCT/GB2000/000769 has identical claims to the previous patent US2002/0005108A1 for what concerns the architecture sensor /controller unit/actuator. Analogously, the function of the processing unit is different from the Control Algorithm. The expired patent US 6,320,113 Bl is similar to the architecture sensor /controller unit/actuator of the patent US2002/0005108A1. It describes a system that provides sound control for an acoustic musical instrument comprising actuators, sensors, and closed-loop transfer functions. A sensor is configured to generate sensed signals based on vibrations from a structure. The sensed signals are input to a controller that processes output signals. The output signals are fed to an actuator that alters the sound of the acoustic instrument. The differences of the present invention compared to this patent are the same as the differences compared to patent US2002/0005108A1. In addition, the controller unit is posed outside the instrument as well as the location of the actuators is different from that of the present invention. The patent US 8,389,835 B2 has identical claims to the previous patent US2002/0005108A1 for what concerns the architecture sensor /controller unit /actuator. However, the controller unit is less advanced than patent US2002/0005108A1 in that it does not seem to be a DSP. The differences of the present invention compared to this patent are the same as the differences compared to patent US2002/0005108A1.

3 Description of the drawings

Fig. 1.1 : is a perspective view of a stringed instrument illustrating its nomenclature.

Fig. 1.2: is a diagrammatic view of the Controlled Instrument showing the location of the sensors and actuators, the separate tonal chamber, and the control system.

Fig. 1.3: illustrates the overall calibration setup for the Cloning Procedure.

Fig. 1.4: is a block diagram of the control system in an augmented instrument. 1.5: is the input/output description of the Target Instrument in the frequency domain.

1.6: is a block diagram describing the functions involved during the optimization procedure.

4 Detailed Description

The method and process of the present invention is described by referring to the figures, which are just an exemplification of the preferred execution and do not limit the invention to just such forms of representation.

The Target Instrument is any instrument composed of a radiating body (i.e., a sound- board (101) and/or a sound box (102)), a bridge (103) , and strings (104) (see Fig. 1.1) . The Controlled Instrument is composed of the following components (see Fig. 1.2): 1. A radiating body (i.e., a soundboard (101) and/or a sound box (102) ), a bridge (103) , and strings (104) . The radiating body is divided into a principal part (201) and a secondary tonal chamber (202) of smaller dimensions that is mechanically uncoupled from the rest of the instrument.

2. A velocity measurement apparatus s (203) , such as an accelerometer or a piezoelet- ric contact microphone, placed in the principal part of the body (201) in a position close to the strings bridge (103) .

3. A force actuator (or other vibratory device capable of providing mechanical ex- citation) a (204) placed in the principal part of the body (201) close to the measurement apparatus s (203) .

4. A force actuator (or other vibratory device capable of providing mechanical exci- tation) <¾ (205) placed in the secondary tone chamber (202) .

5. A generic Central Processing Unit (CPUl) (207) used to implement the real-time control algorithm.

6. A single- channel Analog-to-Digital converter (ADC1) (206) used to sample and quantize the signal coming from the sensor s (203) in order to be processed by the CPUl (207) .

7. A multi-channel Digital-to-Analog converter (DAC) (208) used to drive both the actuators αι (204) , α ₂ (205) from CPUl (207) .

The parameters of the Controlled Instrument are calibrated using a calibration setup that consists of the following items (see Fig. 1.3) :

1. An exemplar of the Controlled Instrument such as described above. no 2. An exemplar of a Target Instrument such as described above, whose acoustic prop- in erties have to be imitated by the Controlled Instrument.

112 3. A pressure measurement apparatus S2 (301) , such as a condenser microphone,

113 placed in the proximity of the instrument to capture the radiated acoustic pressure.

11 4. An Analog-to-Digital converter (ADC2) (302) used to sample and quantize the us pressure signal coming from the microphone ¾ (301) .

lie 5. A generic Central Processing Unit (CPU2) (303) used to run the calibration al-

117 gorithm for deriving the parameters of the control algorithm which runs in CPUl us (207) .

119 4.1 Mathematical Modelling of the Controlled and Target Instruments

120 In this section we characterise the mathematical modelling of the Controlled and Target

121 Instruments, which is the essential preliminary step to specify the Control Algorithm in

122 the next section.

123 Following a classic approach in control systems theory, the input/output relationships

12 of the Controlled Instrument, and the Target Instrument, are detailed respectively in

125 Fig. 1.4. Each solid block (402) (403) (404) (405) (406) (407) represents a frequency

126 response function (FRF) that models a mechanical or acoustical property of the con-

127 sidered instrument, with the exception of the blocks Κ {ω) (404) , Κ.2{ω) (407) that

128 correspond to functions implemented by the digital controller. These functions (404)

129 (407) are the control parameters of the Control Algorithm that we can choose so to

130 have desired vibrations of the Controlled Instrument. The position of both the sensors

131 (203) (301) and actuators (204) (205) are explicated through dashed boxes for clarity

132 purposes, while their input/output behaviour is lumped into the transfer functions that

133 model the physical system.

134 The physical variables explicited on the signal branches are the following: us · U(cj) (408) is a vector whose components are the force signals generated by the

136 N _s strings (409) of the instrument as the result of the excitation of the human

137 player and acting on the bridge that couples the strings to the soundboard.

us · ν{ω) (410) is the velocity of the soundboard in the proximity of the bridge mea-

139 sured by sensor s (203) .

wo · Y _C {D) (411) is the acoustic pressure generated by the principal part of the body

141 instrument as measured by sensor S2 (301) .

142 · Yt ( j) (412) is the acoustic pressure generated by the secondary tone chamber as

143 measured by sensor ¾ (301) .

144 · Y( )) (413) is the total pressure measured by sensor ¾ (301) as the combined

145 action of the principal body and the tone chamber. we The frequency response functions corresponding to mechanical and acoustical parts of

1 7 the Controlled Instrument and the estimation procedures for their parameters are the we following:

149 · H _c (CJ) (402) is the mechanical impedance of the instrument body to the exciting

150 action of the strings. Mathematically it is a row vector of dimension 1 x N _s whose

151 columns are the FRFs from the forces generated by each string and the velocity

152 measured at sensor s (203) . The FRFs of each column can be measured applying

153 the wire break technique (Woodhouse 2004) to string having index n, 1 < n < N _s ,

154 while other strings are temporarily removed from the instrument. The recorded

155 velocity signal measured at sensor s (203) is taken as the impulse response h _CjTl (t)

156 and its Fourier Transform H _Cjn (cj) is the n-th column of H _c (CJ) (402) .

157 · Hi (cj) (403) models the combined action of the response of the actuator a (204)

158 and the mechanical impedance of the instrument body excited in the position of

159 the actuator. It can be measured by feeding the actuator a with a test signal such wo as a Logarithmic Sinusoidal Sweep (LSS) or a Maximum Length Sequence (MLS) , lei recording the output signal measured with sensor s (203) and using standard sys-

162 tem estimation techniques such as, and not limited to, Least Squares Optimization.

163 · A _C {D) (405) models the acoustic radiation impedance of the main body of the

164 instrument. It can be measured by feeding the actuator a (204) with a test signal

165 (LSS or MLS) and recording the output recorded by pressure sensor ¾ (301 ) .

166 · A _t ( )) (406) is a lumped model of the response of the actuator <¾ (205) , the

167 mechanical impedance and the acoustic radiation impedance of the separate tone

168 chamber. It can be measured by feeding the actuator a (204) with a test signal

169 (LSS or MLS) and recording the output recorded by pressure sensor ¾ (301 ) .

170 · Κ {ω) (404) , Κ2 (ω) (407) are the FRFs of two separate computational blocks

171 that represent the digital LTI systems implemented by the algorithms in the CPUl

172 (207) .

173 With basic algebra manipulations, the overall matrix of FRFs G(cj) (502) of the

174 Controlled Instrument system can be derived as

175 The n-th component of G(cj) (502) , referred to as G _n ( )) , models the FRF from the

176 n-th string to the pressure sensor ¾ (301 ) .

177 4.2 Control Algorithm

178 The Control Algorithm is based on a mathematical model of the Controlled Instrument

179 we described in the previous section, which is the acoustic system under control. Such an

180 algorithm is representable as a pair of discrete-time Linear Time Invariant (LTI) systems lei that are specified by Κ {ω) (404) , (407) , respectively, and that are implemented

182 by a microprocessor. The algorithm's behaviour consists of two main parts: (i) the

183 creation of the resonances that are not present among those naturally producible by the

18 Controlled Instrument (i.e. , those resonances that could not be produced without using a

185 Control Algorithm) ; (ii) the cancellation of the resonances that are present among those

186 produced by the Controlled Instrument.

187 Since there are many ways to choose Κ\ {ω) (404) , Κ.2 {ω) (407) , one may have several

188 instances of the Control Algorithm. Each instance has its own implementation complex-

189 ity. Finding a set of parameters for an instance of the Control Algorithm can be seen wo as an optimization problem, where the goal is to minimize in the frequency domain the Mi weighted squared error E (604) between the Controlled Instrument's frequency response

192 G(cj) (401 ) and the desired frequency response D(CJ) (602) . The weighting function

193 W(cj) 605 is arbitrarily chosen using e.g. a psycho- acoustic function that tries to give

19 more importance to the frequency region that are most important for the human ear.

195 In the following, the most general instance is specified (the Multi-Objective Control we Algorithm) , and then some special cases are provided.

197 4.3 Multi-objective Control Algorithm

198 Let D(cj) (602) be a desired FRF, namely a FRF that the Controlled Instrument has

199 to exhibit so that it can produces the desired vibrations. The algorithm consists in spec-

200 ifying the control blocks Κι {ω) (404) , Κ ₂ {ω) (407) so that the FRF of the Controlled

201 Instrument G(CJ) (401 ) , specified in Eq. (4.1) , is as much close as possible to the desired

202 FRF D(cj) (602) . Since the FRF is a matrix of complex numbers, we need to specify

203 in which sense the two FRF are made equal. This is complicated by that we are dealing

204 with complex numbers.

205 Let MD (U) be the matrix whose entries are the modules of the entries of the matrix

206 D(cj) (602) , and let ο(ω) the matrix whose entries are given by the phases of the

207 entries of the matrix D(CJ) (602) . Analogously, let Μς(ω) be the matrix whose entries

208 are the modules of the entries of the matrix G (CJ) (401 ) , and let β(ω) the matrix whose

209 entries are given by the phases of the entries of the matrix G(CJ) (401 ) . Moreover, let

210 II · II be any induced matrix norm, for example 1-norm, 2-norm, Frobenius-norm, oo-norm,

211 max-norm, or min-norm, to mention some of the possibilities.

212 The most general way consists in choosing Κ\ {ω) (404) , Κ.2{ω) (407) by a multi-

213 objective optimisation so that the integral over the frequency domain of a weighted

214 squared induced norm of the matrix difference among MD (U) and Μς(ω) is as small

215 as possible, while the integral over the frequencies of a weighted squared induced norm

216 of the matrix difference among ο (ω) and Φς(ω) is as small as possible. Additionally,

217 this optimisation has to ensure that the choice of Κ {ω) (404) , -¾ (ω) (407) give a

218 FRF matrix G(CJ) must be BIBO (Bounded- Input, Bounded-Output) stable. Otherwise,

219 the resulting system would present self-sustained oscillations due to errors in feedback

220 control, which are usually referred to as "Larsen effect" by musicians. The constraint can

221 be satisfied by exploiting well-known results in control systems theory concerning poles

222 and zero placements of FRF. Formally, we have the following multi-objective optimisation 223 problem:

subject to BIBO stability of G(D) (4.3)

In this optimisation problem, the weighing matrixes W (^) and ΛΥ (ω) are chosen arbitrarily. For example, they can be a psychoacoustic weighting function that gives more importance they can give more importance to the frequencies important for the human hearing system, such as A-weighting, ITU-R 468 or similar functions. The decision variables of the optimisation problem are Κ\ {ω) (404) , Κ.2{ω) (407) . In the problem, -J _MAX is the maximal frequency of interest for the application, usually close to the human hearing frequency limit (e.g. 20,000 / 2 rad/s) . Note that in the optimisation problem we have two objectives: the simultaneous minimisation of the module and the simultaneous minimisation of the phases. The solution of the optimisation problem can be obtained by any solution method for multi-objective optimisation. This should not be a problem, since the solution can be achieved off-line. Nevertheless, in the following, we present some other approaches that are of reduced computational complexity.

A computationally more affordable way to solve optimisation problem (4.2) is via a scalarizarion procedure that leads to a Pareto optimisation as follows: First, we define a secularised cost function weighted by the Pareto coefficient 0 < p < 1:

ρ{Κ {ω) , Κ ₂ {ω) ) =

{P\\ [M _D H - MG(U)] WM M H + (i - ) || [*D M - ΦοΜ] w*H ||} du .

0

(4.4)

236 Then, the Control Algorithm parameters are give by the solution to the following opti-

237 misation problem:

™¾( _ω ), ¾ ( _ω ) ρ(Κι(ω) , Κ ₂ (ω) ) (4.5) subject to BIBO stability of G(CJ) . (4.6)

238 Note that in this optimisation problem, the choice of the coefficient p is left to the

239 implementer. For example, one could even choose p = 1 so to give no relevance to the

2 0 phase minimisation. Alternatively, it can be done by constructing the standard Pareto

2 1 trade-off curve and looking for the knee-point of the courve. Finally, observe that if

242 optimisation problem (4.2) is convex in the decision variables Κ\ {ω) (404) , -¾( ^ω ) (407) ,

243 then the optimal solution returned by (4.5) is identical to the one returned by (4.2) .

244 However, if problem (4.2) is non convex, then the solution of problem (4.5) is a feasible

245 solution for problem (4.2) and in general is sub-optimal for problem (4.2).

246 4.4 M ulti-Objective Sub-Optimal Control Algorithm

247 The methods to determine the values of Κ\ {ω) (404) , Κ.2{ω) (407) can be computational

248 demanding. Here, we describe a sub-optimal method that is of easier implementation. This method is sub-optimal compared to the more general method given by optimisation problems (4.2) and (4.5).

Once a desired FRF D(CJ) (602) is set, the Multi-Objective Sub-Optimal Control Algorithm consists in finding the control blocks Κ\ {ω) (404) , -¾ ( ^Ω ) (407) that simultaneously minimise the squared error between the spectral magnitudes of the controlled and target FRFs, where the errors are defined component-wise:

En W _p ( ) d , 1 < n < N _s , (4.7)

where -J _MAX is the maximal frequency of interest for the application, as for the previous problems, and W _p (CJ) is a psychoacoustic weighting function that gives more importance to the frequencies relevant for the human hearing system, such as A-weighting, ITU-R 468 or similar functions. The optimization is subjected to the constraint that the resulting system defined by G(CJ) (401) must be BIBO (Bounded- Input, Bounded-Output) stable. The optimisation problem is

Ei

E2

m ^m KiH, Κ ₂ (ω) (4.8)

E _n

subject to BIBO stability of G(CJ) (4.9)

261 As for the previous section, the solution to this problem can via standard multi-objective

262 optimisation methods. A computationally simple method of finding a feasible solution,

263 which is optimal if the problem is convex, is by Pareto scalarization, where the solution

26 to (4.8) is achieved by solving the following Pareto optimisation problem m ^m Xi (w), Κ ₂ (ω) PiEi (4.10) subject to BIBO stability of G(u) (4.11)

0 < Pi < 1 Vi (4.12) V Pi 1 . (4.13)

265 Note that in this problem, the choice of the Pareto weighting coefficients _/ ¾ is left to the

266 implementer. Alternatively, one can draw the usual Pareto trade-off curve and choose

267 the pi that give the knee-point.

268 4.5 Multi-Objective Heuristic Control Algorithm

269 Here we propose a simpler method to determine Κ\ {ω) (404) , -¾ ( ^Ω ) (407) . The method

270 is heuristic in the sense that it is not analytically derived from the approach of the previ-

271 ous section, although it is inspired from it. Here, the determination of the values of Κ\ {ω)

272 (404) , -¾ ( ^Ω ) (407) consists of two steps. First, the parameters of the controller -¾( ^ω ) 273 (407) are estimated exploiting that the actuator a ₂ ( 205) is placed in an independent

27 tone chamber. In this way it is possible to reproduce, in the Controlled Instrument, the

275 acoustic resonances that are given by D(CJ) (602) but not in the Controlled Instrument

276 itself when both the actuators are not active. In a formal way, the feed-forward FRFs of

277 the Controlled Instrument when only actuator a ₂ (205) is active, is defined as:

G ₂ ,n M = [Αι{ω)Κ ₂ {ω) + H _c (4.14)

278 In the first step, an optimization problem is posed to find only the controller Κ ₂ {ω)

279 (407) that minimize the weighted target error functions i¾ _n defined as

280 where the exponent /¾ > 1 controls the weight given to the frequency points where

281 the spectral magnitude \D. is larger than \ G _{2 ,} In other words, the error E, t,n

282 is subjected to an additional weight that gives more relevance to the frequency points 283 where target response has resonances that are not present in the Controlled Instrument's 284 response. At the same time, less effort is spent trying to suppress resonances that are present in the Controlled Instrument but not in the Target Instrument. Formally, the optimisation problem posed to determine Κ ₂ {ω) (407) is

subject to BIBO stability of G _{2 n} {u) · (4.17)

287 As for the previous section, the solution to this problem can via standard multi-objective

288 optimisation methods. A computationally simple method of finding a feasible solution,

289 which is optimal if the problem is convex, is by Pareto scalarization, where the solution

290 to (4.16) is achieved by solving the following Pareto optimisation problem

(4.18) i=l

subject to BIBO stability of G ₂ (4.19)

0 < Pi < 1 Vi (4.20)

∑> = 1 . (4.21)

291 Let denote by Κ^ω) the solution to this problem, namely Κζ (ω) is the transfer function

292 of the calibrated controller that drives the actuator a ₂ (205) computed as the result of

293 the first step of the algorithm. 29 If we now use Κ¾ (ω) , the overall FRFs of the Controlled Instrument depends now only on Κ\ {ω) (404) , and results defined as

296 Then, we can determine Κ {ω) (404) by formulating an optimisation problem that

297 minimizes the overall resulting error:

£ι,η (ω) - Ό _η {ω) G W _p ( ) άω, (4.23)

298 where 0 < β _τ < 1 is the exponent that weights the effort towards the suppression of

299 the unwanted resonances in the Controlled Instrument's response and, at the same time,

300 simplify the task of designing a controller that maintains the constraint of BIBO stability.

301 Formally, the optimisation problem posed to determine Κ\ {ω) (404) is

Er,l

Er,2

(4.24) subject to BIBO stability of Gi _jTl (u}) (4.25)

302 As for the optimisation problem of the first step, the solution to this problem can be done

303 via standard multi-objective optimisation methods. A computationally simple method

30 of finding a feasible solution, which is optimal if the problem is convex, is by Pareto

305 scalarization, where the solution to (4.24) is achieved by solving the following Pareto

306 optimisation problem

(4.26) i=l

subject to BIBO stability of G _1>n ( ) (4.27)

0 < a, _t < 1 Mi (4.28)

∑ ^a « ^{= L} (4.29)

307 Let denote by Κ* {ω) the solution to this problem, namely Κ* {ω) is the transfer function

308 of the calibrated controller that drives the actuator a (204) computed as the result of

309 the second step of the algorithm.

310 4.6 Cloning Procedure

311 The Cloning Procedure provides the Control Algorithm with the parameters that reg-

312 ulate actuators of the Controlled Instrument when the Control Algorithm allows the

313 reproduction of the vibrations of the Target Instrument. Otherwise, the Control Al-

314 gorithm can have its parameters set so that the Controlled Instrument can reproduce 315 any desired vibration. The Control Algorithm's behaviour for cloning a Target Instru-

316 ment consists of two main parts: (i) the creation of the resonances that are not present

317 among those naturally producible by the Controlled Instrument (i.e. , those resonances

318 that could not be produced without using a Control Algorithm) , but that are present in

319 the Target Instrument's timbre; (ii) the cancellation of the resonances that are present

320 among those produced by the Controlled Instrument, but that are not present in the

321 Target Instrument's timbre.

322 The objective of the Control Algorithm for cloning a Target Instrument is the imitation

323 of the acoustic properties of a given target acoustic musical instrument by finding a proper

32 set of parameters for the controllers Κ {ω) (404) , -¾ (ω) (407) . A block diagram for the

325 model of the Target Instrument is presented in Fig. 1.5. Input and output variables are

326 the same as for the Controlled Instrument, i.e. , the force vector U(cj) (408) generated

327 at the bridge by the strings and the acoustic pressure Y( i) (413) measured with sensor

328 S2 (301 ) . The lumped matrix frequency response G* (CJ) (502) of the Target Instrument

329 can be estimated by the means of the wire break technique, pulling each string of index

330 n until breakdown and taking the recorded acoustic pressure g _n * {t) at sensor S2 (301) as

331 the n-th impulse response. By taking the Fourier Transform of the impulse responses for

332 each n the matrix G* (CJ) (502) is finally assembled.

333 The Cloning Procedure is then finalised by imposing that D(cj) (602) = G* (CJ) (502)

33 and applying any Control Algorithm of the previous section.

335 4.7 References cited

336 US2002/0005108A1

337 PCT/GB2000/000769

338 US 6,320, 113 Bl

339 US 8,389,835 B2

340 Woodhouse, Jim. "Plucked guitar transients: Comparison of measurements and synthe-

341 sis". Acta Acustica united with Acustica 90.5 (2004) : 945-965.