The present invention relates to a system and method for adapt¬ ing subpicture data for being displayed on a display with a smaller resolution (e.g. a display of a PDA), the subpicture data being temporally correlated with associated video data.

Recently, the use of Personal Digital Assistants (PDAs) became more and more popular. PDAs include small format organizers and often do not have any keypad. Control of a PDA can be accom¬ plished by touching its screen using, for example, a pen.

A lot of different operating systems (OS) are used for PDAs, but the two main platforms are Palm OS from .Palm Inc. and Pocket PC from Microsoft, who both license their system to third party manufacturers. Another PDA operating system which is popular in Europe for wireless devices is Symbian. Pocket PC 2003 is the most up-to-date OS. Features include: Pocket Inter¬ net Explorer, a new Windows Media Player, Network Access, MSN messenger, Infrared Communication, Connection Manager, Enhanced Pocket Outlook, Updated Pocket Word and Excel, and is similar in appearance to a Windows XP desktop.

Common PDAs unite a plurality of functions such as the func¬ tionality of pocket calculators, electronic telephones and address registers, notebooks, PCs and mobile phones.

Another use of a PDA, particularly a handheld PC, is displaying entertainment information such as movies, e.g.., from a DVD. DVD-video data are then displayed on the PDA's screen. PDA screens can be of the type LCD (Liquid Crystal Display) like those found in notebook computers and flat panel displays which can display thousands of colors. A screen's resolution is meas¬ ured by the number of pixels provided horizontally and verti¬ cally (e.g. 640 x 240). The greater the resolution the sharper the image gets. Obviously, resolutions on PDAs are limited by the compactness of the screens. On Palm Units, screens are roughly 4 inches across the diagonal with resolution up to 320 x 240.

For displaying a movie from a DVD on which the movie normally is stored in the MPEG-2 format the DVD-video data is divided and converted into a plurality of smaller data files. The con¬ version may be performed, for example, by "Nero Digital", Nero Digital being a next generation MPEG-4 compatible compression technology by Ahead Software AG, Germany. Nero Digital is com¬ patible with MPEG-4 video and MPEG-4 High Efficiency AAC wherein high quality surround sounds and high definition images for home entertainment are available. Multimedia can be encoded therewith having DVD quality at only 20 % of space. Most popu¬ lar simple and advanced MPEG-4 visual profiles are supported.

In this manner a DVD movie can be transferred to a handheld device, such as a PDA. However, in the future it is expected that DVDs can be playbacked on these devices using a network connection, or a USB type of connection.

Normally, DVD-video data also includes subtitles, e.g. informa¬ tion representing text which is spoken e.g. by actors (which is particularly helpful for hard of hearing persons or persons not understanding the language spoken) .

However, since the geometrical size and resolution of the screens is smaller than the one that the subtitle has been originally been made for, subtitle text displayed thereon is difficult to read if resized with the same ratio as the under¬ lying video. If the resize of the subpicture is done less than the video, the subpicture data might be out of the screen on the left and right side, or becomes predominant in the screen so that it is overlaying a big part of the video.

Therefore, it is an object of the present invention to provide a system and method for adapting subpicture data, particularly subtitle data, which can be displayed on small screens wherein the subpicture is sufficiently recognizable, i.e. has a satis¬ fying resolution.

This object is achieved by a system and method for adapting subpicture data for being displayed on a user device having a small screen, the subpicture data being temporally correlated with associated video data, the method comprising the steps of: reading subpicture data from a source; determining a subpicture data portion from the read subpicture data which has a read¬ ability value less/greater than a predetermined readability limit; and partitioning the subpicture data portion into at least two subpicture data portion parts, and • re-sizing the subpicture data portion parts such that each of the subpicture

data portion parts has a readability value which is greater/ equal or less than the predetermined readability .limit.

Upon employing this method on a conventional data file, par¬ ticularly on a DVD-movie data file, subpicture data which is normally displayed in one or more character lines is converted into a corresponding plurality of single-lined subpictures . Therefore, the geometrical size can be adapted to a normal single line, and even be increased. The (geometrical) display size of the subpicture's information becomes greater, i.e. the resolution becomes better. Consequently the information is easier to recognize.

Preferably, the readability value is determined by the ratio of an area the subpicture data actually overlaps, when being dis¬ played, and a total area the video data takes, when being dis¬ played.

Alternatively, the readability value is obtained by determining a number of character lines, the respective character lines being displayed when the subpicture data is displayed.

This is an easy way to define which subpicture data presenta¬ tion has to be adapted in order to allow a better recognition by a viewer, after this data has been re-sized.

According to a preferred embodiment of the present invention, the step of determining the subpicture data portion includes the step of determining subpicture data which is to be dis¬ played in a number of character lines which is at least two; and determining the number of character lines.

By ensuring that an information which is originally included in two or more character lines is convertable into single-lined character chains, wherein each line of the originally several lines is convertable into a correspondingly re-sized single line, the readability is increased. The content of the original two or more lines does not need to be "pressed" into a single line, which would consequently further lower the display size thereof, but can rather be distributed over a suitable number of single lines displayed subsequently.

Moreover, it is preferred that each subpicture data portion part represents a single character line of the subpicture data portion.

Thus, it is prevented that two of, for example, initially three lines are partitioned into the same subpicture data portion part which then is possibly still not recognizable. That means, for example, for line of a originally three-lined subtitle a single-lined subtitle is generated, resulting in three single lines which are displayed subsequently.

According to another preferred embodiment the step of determin¬ ing the subpicture data portion includes determining a display period thereof.

Thereby one gets the (original) time duration the subtitle data portion originally is to be displayed. This peroid can be used for calculating corresponding display periods of the subpicture data portion parts.

Advantageously, the display period includes a start time and an end time which are temporally correlated with .the associated video data.

The start time determines the beginning of a display of a first subpicture data portion part which belongs to a group of sub- picture data portion parts being obtained when partitioning the corresponding subpicture data portion. The end time determines the last possible time point when the display of a last subpic¬ ture data portion part has to be completed when displaying the group of subpicture data portion parts.

According to a further preferred embodiment, the ^• start time and the end time are assigned to the subpicture data portion parts such that the temporal correlation to the associated video data is maintained.

If the temporal correlation is maintained the subpicture data portion parts can replace the initial subpicture data portion without the need of establishing new temporal associations for a proper display.

According to a still further preferred embodiment, the method further includes the step of determining whether or not an additional subpicture data portion is present, temporally pre¬ ceding or succeeding the subpicture data portion.

If no additional subpicture data portion exists temporally before and/or after a specific subpicture data portion, the display periods of the respective subpicture data portion parts generated from this corresponding specific subpicture data

portion can be shifted into a direction where no additional subpicture data portion is present. Therefore, the respective display periods of the subpicture data portion parts can be extended. This further improves the readability.

Correspondingly, the total display period of all subpicture data portion parts can be greater than the display period of the original subpicture data portion itself.

Preferably, the source is a movie data file, particularly a DVD-video data file, and the file can be stored in a memory of the user device. Moreover, the subpicture data portion parts can replace the corresponding original subpicture data portion.

Therefore, the subpicture data can be displayed in a conven¬ tional manner wherein it is not necessary to change features of the playback device such as a decoder module or the like. The temporal association with the corresponding video data can be maintained, if desired, or be advantageously changed, particu¬ larly extended.

According to a still further preferred embodiment, the step of determining the number of character lines includes applying pattern recognition.

Pattern recognition is one preferred way for determining the number of character lines contained in a subpicture data por¬ tion. The pattern recognition is not dependent on a manual interaction and, therefore, the generation of subpicture data portion parts can be accelerated.

It is to be understood that the features mentioned above and those still to be explained below can be used not only in the combination respectively mentioned, but also in other combina¬ tions or in isolation without leaving the scope of the present invention.

Examples of the invention are shown in the drawings and will be further explained in the following in more detail. In the draw¬ ings:

Fig. 1 is a schematic perspective view of a conventional PDA;

Fig. 2 shows a quantity of subtitle data and its subsets;

Fig. 3 is a flow chart of a first embodiment of the method according to the present invention;

Fig. 4 is a flow chart illustrating a second embodiment of the method of the present invention; and

Fig. 5 is a system of the present invention.

In the following, the term "subpicture data" is used as a ge¬ neric term which includes information such as subtitle, hot spot tinting, overlay graphics, etc.

On a DVD-video disc, for example, subpicture text is stored separately from a main video, usually in the form of low bit- depth (4 color) images. Separately means that the subpicture data exists apart from the main video information, specifically

so that the subtitle can be displayed, or not, at a viewer's discretion.

While the DVD specification (generally MPEG-2) supports up to 32 separate subpicture data streams, any one stream is limited to just 3.36 Mbps bandwidth. To meet the bandwidth requirement as well as to simplify the process of combining them with the main video, subpictures are often composed as run-length en¬ coded bitmaps. Each pixel in such a subpicture image is as¬ signed as a value from a fixed palette of 16 colors.

Each subpicture pixel can be specified as one of four pixel types: background, foreground, emphasis-1 and emphasis-2. Each pixel type can have a different color and transparency value. Additionally, each subpicture pixel's color and transparency can be changed from field to field or from frame to frame. The palette as such can be changed on every program chain (PGC) transition.

Consequently, subpicture data is defined in a very efficient and flexible way that enables, e.g., extensive multi-language or other forms of program augmentation while the data stream is kept down to a manageable bitrate.

There are several ways for final display of the subpicture data on a screen, one of which is hardware mixing.

Hardware mixing involves simply taking, e.g., the subpicture text and placing it on top of the video data. Wherever a sub¬ picture pixel overlaps or overlays a video pixel, it replaces it, completely over-writing the video pixel underneath.

Another way is hardware alpha-blending. Hardware alpha-blending involves blending the subpicture image data with the underlying video data using varying levels of transparency, as dictated by an alpha-channel. This alpha channel contains a mask that de¬ termines how much of the subpicture data is blended with the background video data, and at what percentage.

Fig. 1 shows an exemplary conventional PDA or pocket PC 10.

The PDA 10 includes a display or screen 12 such as an LCD, a housing 14, a menu selector 16 and a memory slot 18. The memory slot 18 can receive a portable memory unit which is not shown in Fig. 1. Video data can be stored in the memory unit.

The PDA 10 can also include an internal memory for storing data, particularly entertainment data such as audio files (MP3, wave files, etc.), video files (MPEG-2, MPEG-4, etc.) and the like.

In Fig. 1, the screen 12 of the PDA 10 displays a movie scene including a two-line subtitle text 20. However, the present invention is not only applicable to small screens of PDAs but also to devices (e.g. smart phones) having small sized screens and in general to all devices with a smaller resolution as the resolution for which the subpicture data has been optimized for.

An entire quantity 1 of subpicture or subtitle data is depicted in Fig. 2.

The quantity 1 includes subpicture data which is to be adapted to be display with a high resolution on a small sized screen, and other subpicture data which does not need to be adapted since its resolution on a small sized screen is already satis¬ factory. The subpicture data to be adapted is indicated with reference numeral 2 and depicted as a subset of the entire subpicture data quantity 1.

The subpicture data to be adapted will be determined in one of the steps of the method according to the present invention, as will be explained below. This subpicture data to be adapted will hereinafter also be called "subpicture data portion". The seam around the subpicture data portion 2 represents data which is not to be adapted necessarily since its resolution is good enough to allow a satisfactory readability thereof on a small sized screen. Preferably the playback device "knows" that no optimization needs to be done.

On the other hand, the subpicture data portion 2, wherein there may also be a plurality (not shown) of subpicture data portions

2, can further be (sub-)divided into a plurality of subpicture data portion fragments or parts 3.

In the following, the relations explained above with respect to Fig. 2 are further explained on the basis of some exemplary subtitles, as they occur e.g. in movies, the movies can be stored on a DVD.

#1 "Hey, Bart! You are not supposed" "to torment Lisa!" "I didn't!! She's annoying ME!!"

#2 "Hasta Ia vista, baby!"

#3 "We are very happy to have you here in our fantastic show"

The first example consists of three textual lines which contain information being actually spoken by movie characters, here by characters of the animated cartoon series "The Simsons", i.e. the text can be heard by a viewer, and simultaneously are dis¬ played visibly on a screen. This first example is a potential subpicture data portion candidate for being adapted according to the method of present invention, as will be explained in more detail below, in order to be displayable with a satisfac¬ tory resolution on a small sized screen.

The first example represents in the light of Fig. 2 a subpic¬ ture data portion 2, which may have a display period of 12 s, for example. This portion 2 can be divided into three subpic¬ ture data portion parts 3, the first being represented by the first line, i.e. by "Hey, Bart! You are not supposed", the second part by the second line "to torment Lisa!", and the third part by "I didn't!! She's annoying ME!!"

As will also be explained in more detail below, these three subpicture data portion parts 3 may be assigned either equal (4s each) or different display periods (e.g. 5s — 2s — 5s) depending on a respective length of each of the parts 3.

The second example may represent a textual single-lined subti¬ tle which does not need to be adapted since it may be regarded as having a sufficient readability.

The third example is also a single-lined subtitle. However, it is much longer in comparison to the second example. Displaying the third exemplary subtitle on a small sized screen may be difficult for a viewer to read. Therefore, even a single-line subtitle can be divided into parts 3. In the case of example # 3, a first part 3 could be represented by "We are very happy to have you" wherein the rest (i.e. "here in our fantastic show!") is represented by a second part 3, both parts 3 being adapted to be displayed as single-line subtitles each having either half of the display period of the original subpicture data portion (i.e. of "We are very happy to have you here in our fantastic show") or any other arbitrary (possible) display period division.

Preferably the resolution change for the subtitles doesn't change throughout the playback, or it changes always for adapt¬ ing to the space available.

Fig. 3 shows a first embodiment of the method of the present invention.

In a first step Sl subpicture data are read or received from a source, e.g. from a DVD-movie disc, for providing base subpic¬ ture data which is to be adapted, if necessary.

In step S2 it is determined from the data read in step Sl if a subpicture data portion is present. As mentioned above, a sub¬ picture data portion is an amount of data which belongs to the read subpicture data, wherein a readability thereof is poor. Subpicture data having a poor readability is selected from the read subpicture data and hereinafter referred to as subpicture

data portion. However, generally not all of the read subpicture data are classified as poor.

Hence, a readability value is determined for each of the read subpicture data. The readability value can be defined in many different ways depending on the type of the subpicture data such as the file format in which the data are stored.

If the subpicture data are stored, for example, as bitmaps, the readability value could be determined by detecting the actual visual overlap percentage of the subpicture data when being mapped on associated video data. For example, if a subtitle is initially displayed with a very small font size, the actual visual overlap percentage, and therefore the resolution, is rather low. This might also happen when, for example, only a single line of subtitle is displayed by a corresponding bitmap. In such a case, a probability that a predetermined readability limit, which can be a predetermined resolution, is remained under is rather high. In this manner, the subpicture data por¬ tion can be detected when the readability value is less than the readability limit.

Alternatively, the readability value can be based on a number of subtitle lines to be displayed (cf. example- #1 above). If the number of subtitle lines is high, i.e. more than one, it is likely that the respective lines become harder to read, par¬ ticularly when displayed on a very small screen such as an LCD of a pocket PC. In this case, the readability limit can be defined as a maximum number of subtitle lines which are (still) allowed to be displayed without the need to change the presen¬ tation thereof. However, in general, already two subtitle lines

being displayed simultaneously are hardly recognizable on a mini-screen.

In step S3 the subpicture data portion is partitioned into at least two subpicture data portion parts . Then these subpicture data portion parts are re-sized. By the step of partitioning the subpicture data portion the amount of subpicture data, which has to be displayed at once, is reduced. ^' This allows to re-size the respective parts to a (geometrical) size (i.e. resolution) which is well recognizable for a viewer.

Of course, this might also cause an adaptation of a display timing. Adaptation of display timing will be explained below.

Alternatively, the so-generated subpicture data portion parts may be stored for repeated playback operations (not shown in Fig. 3).

In Fig. 4 a second embodiment of the method of the present invention is schematically shown.

In a first step S31 subpicture data which are included in video data such as, for example, MPEG-2 data and which are temporally correlated therewith are read into a memory of a data process¬ ing unit such as, for example, a PC.

If the data are provided on a DVD-video disc, the data are normally stored in the MPEG-2 file format. A playback device such as, for example, the PDA 10 of Fig. 1 recognizes DVD-video data by a directory entitled "VIDEOJTS". This directory con¬ tains, in the MPEG-2 standard, all files relevant for display.

The directory VIDEO_TS contains files having one of the follow¬ ing three endings '.1OF', '.BUP, ', and '.VOB'. The '.IFO' ending represents index files. The ¹ BUP' endings represent corresponding backup files. The actual audio and/or video data is contained in files having the endings '.VOB' (Video Object).

VOB files contain audio data, video data and/or subtitle data which are multiplexed. In order to play back, the playback device first has to de-multiplex the data. A VOB file, in turn, also includes a navigation pack (NV_PCK) which contains jump instructions and timing information, and one or more Group of Pictures (GOP) which, in turn, comprise video packs (VJPCK) , audio packs (A_PCK) and subtitle packs (SP_PCK), i.e. the ac¬ tual video/audio and/or subtitle data.

All this information is read in step S31 for allowing espe¬ cially the subpicture data, e.g. the subtitle data, to be ana¬ lyzed. Subtitle data are often stored in the bitmap format '.bmp'. During playback the subtitle bitmaps are overlaid with the actual video data. However, even if the subtitle data are not stored in the bitmap format 'bmp', the method of the pre¬ sent invention can still be carried out. Another subtitle for¬ mat merely influences the way in which the subtitle data are to be analyzed.

In the second step S32, it is determined which of the subpic¬ ture data represents a multi-line subpicture data, e.g. by pattern recognition. Multi-line subpicture data represents, for example, subtitle which is to be displayed in two or more lines. As shown exemplarily in Fig. 1, the subtitle 20 is dis¬ played in two lines. However, since the display 12 of the PDA

10 is very small, e.g. only 4 inches in diagonal, character size becomes really small. The subtitle text 20 is hardly read¬ able.

Thus, in step S32 subtitle data which are displayed in two or more lines are identified for further processing.

In a third step S33 time stamps of corresponding video data, i.e. start time and end time, can be determined. The video data are associated with the multi-line subpicture data which have been determined in the step S32.

Alternatively, only the number of (subtitle) lines of the multi-line subpicture data also can be determined. The number of lines can be used for calculating the display time for each of the single-lined subtitles to be generated.

Of course, both the number of lines and the time stamps can be determined in step S33 in common.

In a further step S34, it can be determined whether or not temporal breaks or interruptions are present. This means, it is determined whether or not further subtitle is to be displayed either temporally before and/or after the actual subtitle data which is determined in step S32. Sometimes several multi-lined subtitles are to be displayed one after the other. In this case, there is no temporal break in the subtitle's temporal stream. Hence, the available time for displaying the respective subtitle is fixed.

If it is determined that no temporal break is present, i.e. there are further subpicture data portion temporally preceding and/or succeeding, the display time of this actual subpicture data portion can readily be divided by the number of charcter lines which is also determined in step S3 for getting the time available for the respective subtitles.

For instance, if the number of a subtitle lines which are to be displayed at once is 3 and the display time for this three- lined subtitle data is 18 seconds, the display time being available for each of the single lines is 6 seconds.

It is also possible to divide the available time in a different manner, e.g. 3s + 4s + 11s or 3s+lsP+4s+lsP+9s (IsP means 1 second pause in between, but could be of course also 5 s pause as example). This decision could be made dependent on the re¬ spective length, i.e. the semantic content, of each subtitle line.

On the other hand, if it is determined in step S34 that there are temporal breaks, i.e. either temporally before and/or after the multi-line subpicture data determined in the step S32, an optimized display time can be calculated in a step S36.

Optimized display time means that it can be, e.g., extended. For example, after a three-lined subtitle phase in a movie there is a subtitle pause in which no information has to be displayed. Depending on the length of this pause the length of display time of one or all the single-lined subtitles, which are to be generated instead of the initial multi-lined subti¬ tle, can be varied.

Again, depending on the semantic content of each single-lined subtitle the display time can be adapted correspondingly.

In a step S37, a plurality of single-line subpicture data is generated using the information obtained in the preceding steps. The geometrical display size, e.g. the character size, of the respective single-line subpicture data can be adapted to the size of a corresponding (original) single-lined subtitle.

If desired, the size can also be increased since merely single- line subpicture data are to be displayed after the method of the present invention has been employed to data containing multi-line subpicture data. Even the size of originally single- line subpicture data can be increased in order to improve the readability of the subtitle.

After the single-line subpicture data is generated, this data can be stored either in the source from which the original data were read or in another memory.

Fig. 5 is a block diagram showing a system 40 according to the present invention.

The system 40 of Fig. 5 can be implemented, e.g., by a PC or the like. It includes a memory 42 into which the subpicture data can be read, for example, from a DVD 43 or any other proper storage medium. The system further includes means 44 for reading the subpicture data, means for determining 46 a subpic¬ ture portion, and means 48 for partitioning and re-sizing. The means 44, 46, and 48 can be implemented, for example, by a mircroprocessor or a CPU 50.

It is to be understood that subpicture data may not only dis¬ play text lines, but also character lines representing menus, navigation icons and the like.

The method of the present invention can be implemented as soft¬ ware and/or as hardware. Therefore, a computer system or a data processing system can be provided, being adapted to carry out the method of the present invention. The invention may also be implemented as a software tool or even in a DVD playback de¬ vice. A computer readable medium may be one of a CD-ROM, DVD, floppy disc, hard disc, etc.

The examples given above for defining the readability value are only of exemplary nature and not intended to limit the defini¬ tion. The readability value can also be defined by the area (in the sense of a bar) a subtitle line takes, when displayed, independent of the actual overlap of the subtitle pixels over the video pixels.

Further, it is apparent that all subpicture data can be adapted in order to obtain a standardized resolution of the subpicture data on small sized screens.

Alternatively, the subpicture data portion parts 3 could also be combined again such that they form a continuous text which runs through the screen, preferably as a horizontal band.

While the invention has been particularly shown with reference to a preferred embodiment thereof, it will be understood by those skilled in the art that various other changes in the form

and details may be made therein without departing from the scope of the invention as defined by the appended claims.