This invention relates to automated or part automated quality control of audio-visual media.

There is a need to ensure that the quality of audio-visual media meets appropriate criteria for its commercial sale and distribution. The criteria are usually expressed in the form of tests that comprise measurements resulting in value that have to fall within thresholds. There are many automated tools that are able to perform computations on the media in order to generate the value for these measurements. These automated measurements are usually calibrated by means of subjective testing or by correlating the results against known good and bad material.

For any given measurement, a particular operation will set its threshold on the value of that measurement to a level that is appropriate for that operation. Operations that make or distribute quality movies are likely to have thresholds that are significantly higher than those that set - for example - by operations that distribute user generated content on the Internet. It is common for higher quality content to be re-purposed for delivery through lower quality delivery channels. For these and other reasons, it would be advantageous if methods and apparatus for quality assessment of audio-visual media could be provided which offer both the flexibility to meet different business needs and the consistency to set standards of quality acceptable across a range of - possibly conflicting - business interests.

There exist a variety of automated quality assessment tools that measure aspects of audio or video quality. However, despite extensive testing by the measurement tool vendors, there still remains significant margin for error in the values resulting from some

measurements. These errors may result in:

• False Positives. These errors result in a piece of material being marked as

acceptable when, in fact, a human operator would have judged the material to have failed the test. An example of this might be a blockiness measure where the level of blockiness is very low, but the position of the blockiness obscures some critical text on the screen. The blockiness measure would not have triggered because the measure itself did not take into account that certain parts of the screen were more critical to the viewer than others.

• False Negatives. These errors result in a piece of material being marked as a failure when, in fact, a human operator would have judged the material to have passed the test. An example of this might be a blockiness measure applied to a music video where a special "blockiness" effect was applied to the picture for creative reasons. The blockiness measure would be triggered, but the measurement tool has no independent way of knowing that blockiness was part of the creative intent.

• Differing results from different measurement vendors. Most of the automated

measurements used in the industry are not standardized to a level where consistent pass/fail results are possible between different vendors. Many measurement vendors adapt their measures to include extra intelligence, filtering, motion compensation or other techniques to gain an edge in the market place. This means that it is very difficult for a media company to compare results from vendors without using expensive and highly trained staff to interpret results. In a high volume "media manufacturing" environment, this is often impractical.

• Increased failure rates from the creative-intent of the media. Media files are, by

definition, intended for human consumption as entertainment. Audiences like to be surprised, so media often contains content that surprises humans and measurement tools. For example a sequence that becomes black and white for a brief period would trigger a "black and white" measuring tool. In a media file that contained video shot in colour, this would be a surprising result and should be categorized as an error unless specific knowledge about the media was communicated to the measurement system. This knowledge would for example mark "black and white content is ok between 10 minutes and 15 minutes in the file".

Whilst the disadvantages of any specific automated measurement tool can usually be overcome through operator involvement, it is not viable economically to involve operators at each stage of an audio-video delivery chain or within each involved business.

It is an object of embodiments of the present information to overcome or ameliorate some or all of these disadvantages.

Accordingly, the present invention consists in one aspect in apparatus for monitoring the quality of an audio-video process chain comprising a plurality of processes operating in stages on audio-video content, the apparatus comprising a plurality of measurement tool interfaces for connection with measurement tools to receive measurements; and a confidence threshold processor for converting each measurement to a threshold

confidence; wherein the confidence threshold processor receives said measurement, a threshold value of acceptability; a latitude of acceptability relating to said threshold; and an error function relating to the reliability of the measurement; said confidence threshold processor being configured such that the threshold confidence extends linearly between a value denoting certainty of acceptability and a value denoting certainty of unacceptability. A graphical user interface may include a linear graphical representation of each threshold confidence.

In another aspect, the present invention consists in a method of controlling an audio-video process chain comprising a plurality of processes operating in stages on audio-video content, the method comprising the steps of: taking in respective measurement tools a plurality of measurements; in a processor, converting each measurement to a threshold confidence by normalizing where required and comparing with a threshold value of acceptability taking into account the latitude of acceptability associated with that threshold and an error function relating to the reliability of the measurement, such that the threshold confidence extends linearly between a value denoting certainty of acceptability and a value denoting certainty of unacceptability; defining pass and fail ranges of values of the threshold confidence; storing for delivery or further processing audio-video content having a threshold confidence within the pass range; and rejecting audio-video content having a threshold confidence within the fail range. A warn range of values of the threshold confidence may be defined between and contiguous with said pass and fail ranges and content having a threshold confidence within the warn range may be diverted for reworking.

In another aspect, the present invention consists in a method of monitoring the quality of an audio-video process chain comprising a plurality of processes operating in stages on audio- video content, the method comprising the steps of taking a plurality of measurements M _{g S} where #g denotes the measurement group, #n denotes the measurement tool employed in taking the measurement and #s denotes the processing stage at which the measurement is taken; converting each measurement M _{g S} to a threshold confidence TC _g , _n , _s by normalizing where required and comparing with a threshold value of acceptability taking into account the latitude L _{g s} of acceptability and an error function EB _{g S} relating to the reliability of the measurement M _{g S} such that the threshold confidence TC _{g S} extends linearly between a value denoting certainty of acceptability and a value denoting certainty of unacceptability.

Preferably the method further comprises providing at a first process stage for an operator viewing the audio-video content to input an Override Confidence OC _{g S} to override the threshold confidence TC _{g S} ; and, at a second subsequent process stage, electing automatically whether to override the threshold confidence TC _{g S} with the previous Override Confidence OC _{g S} based on the second stage measurement M _{g S} . In the case of an operator inputting an Override Confidence OC _g,n,s , the overridden threshold confidence TC _g,n,s and its associated normalised measurement value N _{g S} may be made available in subsequent process stages.

Pass and fail ranges of values of the threshold confidence TC _g,n,s may be defined. The method may suitably comprise the steps of approving and/or storing for delivery or further processing audio-video content having a threshold confidence TC _g,n,s within the pass range and rejecting audio-video content having a threshold confidence TC _g,n,s within the fail range.

Pass, warn and fail ranges of values of the threshold confidence TC _{g S} . may be defined. The method may suitably comprise the steps of approving and/or storing for delivery or further processing audio-video content having a threshold confidence TC _g,n,s within the pass range; diverting for reworking audio-video content having a threshold confidence TC _g,n,s within the warn range and rejecting audio-video content having a threshold confidence TCg,n,s within the fail range.

A collection of audio-video content may be ranked in accordance with the threshold confidence TC _g,n,s and reworking resource is allocated to audio-video content within the collection in dependence upon said ranking.

TCg,n,s is represented in the embodiment of this invention by a number between 0 and some maximal value (for example between TC _min = 0 and TC _max = 100%). The value 0 represents certainty that the measured value represents an error and TC _max represents certainty that the measured value represents a pass. Calculation of values of TC are system dependent but can be represented by the following representation:

TC Cg (N _{g S} , EB _{g S} , TV _{g s} , L _{g s} )

where N _{g S} is a normalised measurement value, TV _{g s} is a threshold value denoting pass/fail, EB _{g S} is an error bar function that denotes the statistical uncertainty of that measurement type and C _g is a confidence mapping function that converts normalised measurement values of the given measurement type to a linearly, monotonic threshold confidence value based on that measurement's properties.

The latitude L _{g s} denotes the tolerance of a measurement to the threshold and may vary between process stages. Threshold confidence TC _g,n,s values for all tools in the same measurement group may be aggregated at each process stage. The result is an overall threshold confidence for that group. An example of this is the application of different loudness tools to a piece of music that has undergone processing. The answer the business question "Is the music too loud and has the music always been too loud?" can be answered by inspecting the threshold confidence for a group of loudness measurements spanning the processes the music underwent in a facility.

The present invention consists in another aspect in a method of monitoring the quality of an audio-video process chain comprising a plurality of processes operating in stages on audio- video content, the method comprising the steps of taking a plurality of measurements comprising at least a first measurement at a first process stage and at least a second measurement at a second process stage measurement; generating for each measure an error function which is monotonically related to the expected error of the measurement and combining the error function with the measure to form a measurement range; providing at a first process stage for an operator viewing the audio-video content to override the first measure with a first operator override; and, at a second subsequent process stage, automatically overriding the second measurement with the first operator override if the second measurement range overlaps the first measurement or overlaps the first operator override.

The present invention consists in still another aspect in apparatus for monitoring the quality of an audio-video process chain comprising a plurality of processes operating in stages on audio-video content, the apparatus comprising a plurality of measurement tool interfaces for connection with measurement tools to receive measurements M _{g S} where #g denotes the measurement group, #n denotes the measurement tool employed in taking the measurement and #s denotes the processing stage at which the measurement is taken; and means for converting each measurement M _{g S} to a threshold confidence TC _g , _n , _s by normalizing where required and comparing with a threshold value of acceptability taking into account the latitude L _{g s} of acceptability and an error function EB _{g S} relating to the reliability of the measurement M _{g S} such that the threshold confidence TC _{g S} extends linearly between a value denoting certainty of acceptability and a value denoting certainty of unacceptability.

The apparatus may further comprising an operator override input enabling an operator viewing the audio-video content to input an Override Confidence OC _{g S} to override the threshold confidence TC _{g S} and a graphical user interface including a linear graphical representation of the measurement M _{g S} and/or each threshold confidence TC _{g S.} .

In the context of this invention the term audio-video content includes audio content; video content and audio-video content. It will be seen that some embodiments of the invention provide a mechanism for storing and automatically propagating operator derived information in a way that enables the results of a media measurement tool to be trusted to a greater extent than hitherto.

The present invention will now be described by way of example, with reference to the accompanying drawings, in which:

Figure 1 is a block diagram illustrating an embodiment of the invention:

Figures 2 and 3 illustrate the lifecycle of a measurement;

Figures 4 and 5 illustrate the accumulation of measurement information as processes are applied to a piece of media;

Figure 6 is a block diagram similar to Figure 1 , illustrating a further embodiment of the invention;

Figure 7 illustrates multiple measurements at multiple stages of the lifecycle of some media;

Figure 8 illustrates an enhancement on Figure 7 that takes into account measurement inaccuracies;

Figure 9 illustrates the handling of operator overrides; and

Figure 10 is a block diagram illustrating constituent parts of a system according to one embodiment of the invention.

Figure 1 is a block diagram illustrating a simplified arrangement according to the present invention. The skilled man will understand that the functionality depicted in Figure 1 can be implemented in a wide variety of forms including dedicated hardware, programmable apparatus; software and appropriate combinations of the foregoing.

Audio-video content 100 is delivered to a processing block 101 providing process PR _P. The audio-video content or asset 102 is then passed for storing or further processing or failed, in decision block 120.

A measurement tool 103 takes a measurement on the audio-video content 102 and that measurement is processed in blocks 103 to 106 (the functionality of which will be described in detail below) to control the decision block 120.

It will be helpful to define some terms used in the diagrams and texts:

PR _P Process #p which operates on audio-video material MTn Measurement Tool #n that comprises a number of discrete measurements, e.g. MT _A might be the Acme Testing Tool.

Mg n.s Measurement performed by tool #n of measurement tool group #g at

processing stage #s of a workflow, e.g. M _3A1 and M _3A2 and M _3A3 are all the same type of measurement created with the same tool, but performed at different stages in the lifecycle of the material.

Ng,n,s Normalised Measurement in group #g from tool #n at stage #s.

TV _g s Threshold Value for group #g at stage #s. This is the critical value that a

Normalised Measurement must meet for a measurement to have "passed". TCg,n,s Threshold Confidence of Measurement #g implemented by tool #n and used at stage #s

Lg s Latitude in the Threshold Value for group #g at stage #s. This controls the range of Threshold Values for which a Normalised Measurement will generate non-zero Threshold Confidence values. EBg n.s Error Bar function. A factor that controls the size of the error bars of a

measurement in group #g of tool #n at stage #s.

OCg,n,s Override Confidence of a Measurement in group #g from tool #n at stage #s.

This is the Confidence value given to an Override whether introduce manually by a human at a GUI or automatically via stored knowledge in, for example, a database.

ACg,n,s Auto-assigned Confidence of measurement in group #g, from tool #n at stage #s.

An explanation of the lifecycle of a measurement may be helpful in clarifying the

relationship between the different terms is clear. Referring to Figure 2, a measure of video blockiness is taken before and after a transcode process PRi . The measurement is being performed by a tool of the video-blockiness group MT _B . This group is identified in this example by the B subscript. Tools are categorised so that any tool within a group may have its Normalised Measurement Value compared with any other tool's Normalised Measurement within the group. Define measurement M _B, x _, o at stage 0 before Process PRi with tool MT _X from Group _B .

Define measurement Μ _Β, γ _, ι at stage 1 after Process PRi with tool MT _Y from Group _B . These measurements cannot be directly compared unless their units are normalised by some tool specific function that ensures their units, range and linearity are similar.

N _B ,x,o = fx (M _B ,x,o) and N _B , _Y ,i = f _Y (M _B , _Y ,i)

Where f _x () and fy() are the tool specific normalisation functions for Tools MT _X and MT _Y . Now that there are comparable results, it is possible to compare the Normalised Value N _{g S} to a Threshold Value TV _{g s} to see if the measurement was within tolerance at that stage of the media's lifecycle. For simplicity in Figure 2, the Threshold Values have been made identical and are shown as TV _B .

It is necessary to take into account the fact that many media measurements are inherently uncertain so there is a need to know the confidence that a result has exceeded a threshold. It should be noted that some Threshold Values are minimal in nature (e.g. to pass a test, all 8 bit pixel values should be greater than the decimal Threshold Value 16) and some Threshold Values are maximal in nature (e.g. all to pass a test, all 8 bit pixel values should be less than the decimal Threshold Value 235). The examples in this text will predominantly use the maximal threshold case. The same principles hold for the minimal threshold case.

There are two main elements of uncertainty that are taken into account in this method. The first is the Latitude L _{g s} of a threshold in group #g at stage #s of a measurement. This property measures how much uncertainty can be tolerated at stage #s in the media lifecycle and is shown in Figure 2 by the fine dashed line above the maximal Threshold Value TV _B . In a simple system with a maximal Threshold Value, a Normalised Measurement N must be below some threshold to pass. From the example in Figure 2, it can be seen that both the normalised measurement values are above the maximum Threshold Value and that they would both constitute a failure.

In a media system, many of these absolute Threshold Values are in practice somewhat arbitrary, so the present invention employs the concept of Threshold Confidence TC _{g S} . The Normalised Measurement Values N _{g S} all have the same units, range and linearity so that they can be compared with other normalised measurements in the same group #g. The Threshold Confidence TC _g , _n , _s is a unitless value ranging from 0 to some maximum value TC _max that expresses the confidence that a measurement N _{g S} passed a threshold value TV _{g s} . Because TC _g , _n , _s has been generated as a linear value with a define range and no units, it can be compared and manipulated with dissimilar measurements from different groups. Importantly, use of the instrument TC _g , _n , _s allows more accurate QC results to be obtained in a QC process by propagation and aggregation of TC _g , _n , _s from measurements that could otherwise not be compared.

A worked example could be to calculate the TC for the whitest pixel that is allowed in a picture. In an 8 bit video system, black pixels have the value 16 and white pixels have the value 235. Checking the white value constitutes a Measurement Value with no intrinsic measurement error where the threshold Value TV _{g s} is a maximal threshold (235), one way to calculate the Threshold Confidence TC _g , _n , _s would be to calculate the percentage distance of the Normalised Measurement in the Latitude window (values 233-237):

TC _g ,n,s= 100% for all N _g , _n , _s < TV _g , _s - L _g , _s i.e. less than 233 TC _g ,n,s= 0% for all N _{g S} > TV _g , _s + L _g , _s i.e. greater than 237

TC _g ,n,s= (Ng,n,s - Vg.s)/ 2L _g , _s for all other values of N _{g S} i.e. a linear percentage of the distance the measurement lies from the edges of the threshold.

This is shown in Figure 2, where both normalised measures are above (TV _B + L _B ) and they would therefore have a TC value of 0% - corresponding to a fail.

The Threshold Confidence, however, becomes more complicated due to the fact that many measurements (such as blockiness) are intrinsically inaccurate and subjective. Others (such as loudness) use Normalised Measurement units that are non-linear in nature. To compensate for this, an Error Bar function is introduced to allow for a window of possible values that might have been made by a given tool. In general, the Error Bar function will be a simple linear multiplier, but may be more complex for some measurement groups such as blockiness where other picture detail may be taken into account.

The Error Bar Function is shown pictorially in Figure 2 as an I beam centred around the measurement value. Once the Error Bar has been taken into account it can be seen that there is a chance that the first measure M _B, x _, o might actually have passed, whereas Μ _Β, γ _, ι is still a definite fail.

For the remaining examples in this text, we will use the abstract function C _g to create a Threshold Confidence value TC _g , _n , _s for measurement group #g. This function is different for each measurement group and takes into account differences in output between different measurement tools of the same measurement group as well as its intrinsic error properties when generating TC. Figure 3 illustrates different levels of complexity in the function C _g . Figure 3a shows the simplest example for a measure with a maximal TV. It shows the mapping outlined in the text above where the TC is a linear representation of the distance that N _{g S} is from TV _{g s} .

TCn,g Cg (Ng _{n s} , EBg _{n s} , TVg _s , Lg s)

Figure 3b shows a more realistic mapping function C _g that takes into account the fact that the TV _g s threshold is not symmetrical and that values of N that are less than the TV _{g s} are not 100% certain to be passes due to the fact that the TV _{g s} is a somewhat arbitrary number such as would be the case for a blockiness threshold.

Figure 3c shows the effect of the £B _{g n s} () function. It has the effect of adding a non-linear transfer function around the actual value of N.

This unitless measure of threshold confidence for a measurement may now be compared with other TC _g,n,s values from different groups #g made with different tools #n to give an overall confidence that material has passed or failed at any stage #s in its lifecycle. The TC _g, n _, s value also acts as a parameter for allowing an override confidence to auto-propagate from an earlier stage of a Quality Control process to a latter stage of a Quality Control process.

An example of the process for using these values is given in Figure 4. This shows two processes (Process PF^ and Process PR ₂ ) operating sequentially on the same audio-video content. The first step is to perform a set of measurements on the incoming content and to create Report R ₀ . This report contains the results from Tool MT _A that generates three measurements in groups X, Y, and Z to provide Measurement Values M _XA0 , M _YA0 and ΜΖ _, Α _, Ο- Tool MTA can be regarded as producing the initial QC report. It is of course a single ended measure.

After processing the media in Process PRi the output media is tested again, as shown in Figure 5. In this case, two tools are used. Tool MT _A is used to create Report Ri and Tool MTB is used to create Report R ₂ . For the case of Tool MT _A the same three measurement techniques are used to create new Measurement Values M _XA1 , M _YA1 and M _ZA i - Tool MT _A can perform a comparative or double ended measure. Tool MT _A 's new QC report and Tool MT _B 's QC are appended to the original QC report R ₀ .

After processing the media in Process PR ₂ the output media is tested again. In this case three tools are used. Tool MT _A is used to create Report R ₃ , Tool MT _C is used to create Report R ₄ and Tool MT _D is used to create Report R ₅ . For the case of Tool MT _A the same three measurements techniques are used to create new Measurement Values M _XA3 , M _YA3

It is helpful here to refer to Figure 6 which takes the same diagrammatic approach as Figure 1 but illustrates measurements being taken at more than one process stage. A Threshold Confidence provided by the measurement tool 93 operating on content prior to the process PR _P ( with processing blocks 93 to 96) is grouped with the current Threshold Confidence in TC Grouping block 130.

Figure 6 also shows an auto-propagation block 140 which also receives an operator override input from block 150 and a multi-tool combination block 160, the functionality of which will be described in more detail below.

To exemplify comparison between the same tool from the same group at difference stages, measurements from group X are first normalized with a measurement specific process as follows:

The next step is to introduce Threshold Confidence using the measurement specific mapping function C _g .

TCx,A,1 ⁼ C _g (Nx A,1 , ΕΒχ Α.1 , TVx i, l_x i)

TCx,A,2 ⁼ Cg, (Νχ _Α 2, ΕΒχ Α,2, TVx 2, Lx^)

TCx,A,3 ⁼ Cg, (Νχ _Α 3, ΕΒχ Α,3, TVx s, ί,χ β)

Typically the resulting values of TC _g,n,s will correspond to some business indicator:

• Pass / Fail - for processes that have simple rejection workflows (TC greater than some threshold) · Pass / Warn / Fail - For more complex procedure where a supervisor might be

required to judge rework on an intermediate state. For example

o Pass TC > 66%

o Warn TC >33% and TC<= 66%

o Fail TC<= 33% Figure 7 shows five stages of measurement of an abstract value N and shows the mapping of those values of N into TC. Note that the (somewhat arbitrary) setting of a pass to be 66% maps back into N space as a staggered line represented by the dash-dot central line.

The present invention introduces the concept of threshold confidence where, for example, a continuous value from 0% to 100% is introduced that allows any measure to have a continuous scale of confidence that the measure has passed the test.

• The value 100% corresponds to absolute certainty that a particular measure has passed a test.

• The value 0% corresponds to absolute certainty that a particular measure has failed a test.

• Any value between 0% and 100% gives the confidence with which a measure has passed the test.

Threshold Confidence varies according to the measurement being carried out. Here are some examples:

1. "Sub-blacks" and "Supra Whites". These measures correspond to Luminance

values of a TV signal that have exceeded the legal values from a Rec.601 video signal. The thresholds are 16 and 235 for 8 bit video. A value of 15 or 236 would be given a TC of 0% whereas a value of 17 or 234 would be given a TC of 100%. A value of 16 or 235 would be very, very close to failure without actually failing.

Operationally, this would be given a TC of around 60% to indicate that it was very close to a failure threshold.

2 "Loudness" measurements are a single floating point value that indicates a

modified mean loudness over a whole media clip. If this modified mean loudness exceeds a given threshold then it is a clear fail and would be allocated a TC of 0% If the mean was less than the legal threshold mandated by either the government regulator or broadcaster. Then a TC value of

100% - some_function(spread(loudness), safety_margin) would be allocated. The closer that the mean was to the threshold, then the lower the TC. If the spread of loudness values was very large, then this increases the uncertainty that the mean loudness is truly representative of the clip, reducing the confidence again.

3 "Blockiness" measurements are often a single floating point value that represent the perceived worst blockiness in a clip. The TC value can be set according to an overall blockiness measure and a ratio of how detailed the imagery in the clip was, when the blockiness measure was high. The more detailed the video content, the less visible that blockiness can be.

Threshold Confidence on its own is a useful, new concept that considerably strengthens the use in a system of automated QC measures.

It is however in the nature of many audio or video applications that the creation of interesting and entertaining content pushes the media in unexpected directions. These unexpected departures from what previously had been regarded as normal may be interpreted by automated QC measures as failures. It is also the case that, however well designed the automated QC measures, a skilled operator may reach a different and better conclusion. For example, the enormous variation in styles and genre of video content passing through a typical delivery channel, makes it very difficult for an automated QC measure to distinguish reliably between what an end user would perceive as acceptable and unacceptable departures from a perhaps arbitrary measurement threshold.

In a QC workflow according to the embodiment shown in Figure 6, a human operator can optionally override the results of an automatic tool. An operator can for example take a result that was deemed to be a failure by the automatic tool and reclassify it as a pass by assigning an Override Confidence value.

Figure 8 shows an example where an operator decided that the measurement at stage 3 was not, in fact, an error, but was a pass. The operator was around 70% confident that it was a pass and this value OC _g , _n , _s is now taken as the new threshold confidence for that particular measurement. The calculated TC _g , _n , _s has been overridden.

If an operator overrides the value of a measurement, then all user interaction (GUI, reports etc) will display the Override Confidence rather than the Threshold Confidence. Override Confidence uses the same scale as TC and indicates how confident the operator was that the measurement was a pass. In many cases operational staff will only ascribe values of 0% or 100%.

For some measurements where human judgement is vital (such as a blockiness measure), the operator would be presented with a user interface to allow them to give a true OC value between 0% and 100%. This allows the operator to give a value that represents "how close to being fit for purpose" the media clip is. This threshold will vary based on the value of the material and the target to which the material is being sent. Operators are trained to make these judgements. A rule for automatic propagation of Override Confidence OC _g , _n , _s is based on the normalized measures and normalized confidences. Figure 8 shows an example where an operator overrode the TC of measurement 3, and there is applied the rule "If the measurements are still consistent then propagate the override confidence automatically". This method has the advantage that a single override can be used automatically in several stages of the lifecycle of the material.

To allow Auto-propagation, it is desirable to define the word "consistent". There is scope for tuning these rules for each specific measurement, but a useful basic rule is that if the TC is the same or better (higher or lower depending on the nature of the threshold) than the previous stage then flag it as a valid override. Whether or not the override is actually propagated may be determined by the product configuration to maximise flexibility.

As a worked example the False-Negative override case (OC _g , _n , _s > TC _g , _n , _s ) - Figure 8 may be auto-propagated with the following sample rules:

If [( (TCg.n.s >= TC _g ,n,s-i ) or (Ng, _n ,s-i falls within EB _g ,n,s(Ng, _n ,s)) )

and (OC _g , _n , _s -i > TC _g , _n , _s -i )] then AC _g , _n , _s = OC _g , _n , _s -i

i.e. if the TC is the same or higher than the previous stage then flag it as a valid override.

False-Positive override case (OC _g , _n , _s < TC _g , _n , _s ) - Figure 9:

If [( (TCg.n.s <= TCg, _n ,s-i ) or (N _g ,n,s-i falls within ΕΒ _9ι „ _ι5 (Ν _9ι „, ₅ )) )

and (OC _g ,n,s-i < TC _g , _n , _s -i )] then AC _g , _n , _s = OC _g , _n , _s -i

i.e. if the TC is the same or lower then flag it as a valid override.

An enhancement to these basic sample rules is to look back over all previous stages and not just the most recent stage. This would create a control that would allow "auto-propagate overrides from any previous stage".

A further enhancement is to provide that, if the first rule is not met, but there is an overlap between the TC error bars of the current stage and the TC error bars of the stage where the override was created, then this should be flagged as "likely to be a valid override". This catches the case where it is difficult to ascertain that the error bars were either masking a bad value or degrading a good value.

When the auto-propagation rules are applied and there exists an Override-Confidence, there can be up to 3 confidence values. They are applied in this order of precedence:

1. OCg,n,s - this is used if it exists

2. ACg,n,s - this is used if it exists and the business rules allow 3. TCg, _n ,s - the default value of threshold confidence for this measure

Using this simple rule and this methodology, operator actions can be rapidly and

automatically applied to many different tools taking into account their inherent differences and algorithms.

Multi-Staqe, Single-Tool UQC

Using the same methodology as above, but looking across difference stages of a group of measurements, the same rules can be applied to propagate an upstream decision of an operator to a downstream measurements and gives the ability to propagate judgements in an environment where allowances are made for the uncertainty of the underlying measures. · When an identical measure is used at different stages of the lifecycle, the Latitude Lg _s may be derived from an upstream stage L _{g s} -i to reflect business rules. For example

o "Sub Blacks" and "Supra Whites" - The final value of L _{g s} would be set 0% for broadcast delivery because there is no tolerance allowed in this measure for final delivery. Upstream, however, the value could be set to a relatively large value like 5% because moderate excursions during the signal processing path can lead to improved image quality.

o "Loudness" - The final value of L _{g s} for broadcasting would be set to a very small value because there are strict rules for Loudness in a broadcasting chain, whereas for internet delivery L _{g s} would be set to a high value on the grounds that there are few loudness requirements and the listening environment is very different to broadcast.

o "Blockiness" - The final value of L _{g s} would be increased (e.g. L _{g s} = 2 x L _{g s} -i) at the final delivery because a small amount of blockiness is often tolerated at final delivery, whereas blockiness is rarely tolerated upstream during the program creation process.

Multi-stage, Single Tool, Reworked QC

The methodology can also be used in conjunction with propagated upstream

measurements and decisions where an adjustment rework operation is performed to either improve on a failing measure, or to ensure acceptance of the result within a tighter specification. This continuation of the lifecycle has the added advantage of automatically verifying that repeated application of the original measurements produce results within the expected Latitude range, i.e. that fixing problem A has not created a problem B. Overrides in this case may be created, not just by operators, but also from stored knowledge about the content. For example a low chroma detector can be automatically overridden if it is known that certain portions of the content are in black and white rather than in colour.

Multi-Staqe, Multi-Tool

The rules and values can be applied without change to multiple tools at multiple stages.

Figures 8 & 9 could also apply to the use of different tools (with the "n" subscript changed for a different subscript as appropriate). Using this aspect of the present invention, different tools can now be compared in a way that was not previously possible.

The described methodology allows the threshold for individual QC measurements to remain independent whilst simultaneously being able to combine and weight dissimilar

measurements types to give an overall confidence that media is of acceptable quality for an application. Typically the thresholds for a given measurement will be determined by one or all of the following drivers: · Hard limits determined by some delivery specification e.g. Black level of 16 in an 8 bit system

• Hard limits determined by some Service Level agreement e.g. exactly 5 minutes of black frames every 25 minutes.

• Hard limits determined by a specification that gives variable results depending on start point or measurement window, e.g. program should have a loudness of -24dB

• Soft limits determined by some subjective criteria, e.g. no visible blocking

• Soft limits determined by some performance criteria such as the complexity of a bitstream and the likelihood of some parameters to cause a decoder to fail e.g. too many large motion vectors, very long runs of zero bytes or motion vectors that cause off-screen image information to be fetched.

No single measurement may cause a piece of material to fail, but the described methods can be used to show that the overall confidence of a pass may be very low.

As illustrated in Figure 10, a platform 390 is provided between two process stages, with like platforms being provided between other process stages. Preferably, a platform is provided before or after each significant process or collection of processes in a process chain. Often, a platform will be provided at the point of first receipt of content by a commercial entity within a chain or group of entities working together in the creation; post-production; repurposing or other processing (where appropriate) and delivery of content, In that case, a threshold confidence value can determine whether or not content is accepted and/or whether content is stored or not.

Platform 390 comprises a unifier block 391 which communicates with the various

measurement tools through respective measurement tool interfaces 932. There is also provided a graphical user interface. Inputs 394 and 395 communicate with upstream and downstream platforms. An operator override 396 is also provided. Operation of the platform will be understood from the foregoing description.

Separation of the measurement tool interfaces 392 from the unifier block 391 simplifies the extension of the platform to work with new proprietary measurement tools.

The GUI will preferably take advantage of the linear, dimensionless nature of the threshold confidence to provide absolute or relative indications of quality which can be interpreted instantly, even by non-specialists. These may take the form of simple bars, although the linear nature can still be maintained with curved lines. In the typical case, where the range of threshold confidence values is divided into pass and fail ranges or pass, warn and fail ranges, the simplicity of the interface can be maintained by changing the colour of the linear bar (curved of straight) to signify movement from one range to the next. The colours green, yellow and red can conveniently be employed for the pass, warn and fail ranges, respectively.

Content which has a threshold confidence value in the warn range can be treated in various ways. It may be appropriate automatically to divert any such content to reworking station where (usually) human operators can attempt to improve the quality. If the reworking resources are limited, the actual threshold confidence values can be used to rank the content to provide a priority order for reworking.

It will be understood that the process chain has been depicted symbolically and "adjacent" processes may actually be widely separated geographically and in time. It will be convenient to transport the reports which have been described above, in the form of metadata associated with the audio-video content.

The platform may take the form of software only, in a case where appropriate general purpose hardware already exists with the appropriate connectivity. Hardware/software hybrids or wholly hardware solutions are of course also possible.