Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4180 Silvia.Pfeiffer@csiro.au http://www.ict.csiro.au/Silvia.Pfeiffer/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4222 Conrad.Parker@csiro.au http://www.ict.csiro.au/Conrad.Parker/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4222 Andre.Pang@csiro.au http://www.ict.csiro.au/

This document specifies a syntax for addressing time intervals within time-based Web resources through URI queries and fragments. This scheme is particularly used for Annodex and CMML Web resources, though it could be picked up by any other time-based Web resource format. Temporal addressing enables, e.g., direct access to a clip inside a video stored on a Web Server, or direct jumps to a specific event within a piece of music. The syntax is not restricted to audio or video Web resources though, but covers all kinds of Web resources that contain time-continuous information. When a server (e.g. a Web Server) supports the URI query syntax, it is able to extract the given time subsections from the base resource and serve them as another resource of the same type. Specifying a time point or interval through a URI fragment in contrast does not deliver a subpart of the resource - instead it is up to the user agent and the resource type as to what action is invoked. Examples are a fast forward to a time point, or a zoom into a time interval. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Many resources on the World Wide Web represent information that relate to a timeline of events. Time-sampled recordings of sound, temporally interleaved audio-visual streams, or data files containing time series data are examples. This document describes a syntax for addressing temporal offsets and time intervals in such resources. The aim is to make it simple to incorporate infrastructure into the Internet which supports the addressing and retrieval of temporal subparts of time-based Web resources, as well as the automated processing of such subparts for reuse. The temporal URI interval specifications are to be used on resources that represent a specific timeline. An example of such resources are most resources of MIME types "audio/*" and "video/*". Files containing commands for creating time-continuous experiences (such as SMIL files for authoring interactive multimedia, or VRML files for authoring 3D worlds) cannot be addressed in this manner. However, if one particular user interaction (i.e. one experience) is recorded, the recording contains data that relates to a single timeline and can be addressed with the manner described in this specification. The temporal URI queries and fragments specified in this document have been implemented and demonstrated to work with Annodex and CMML format Web resources over HTTP. A possible implementation over RTP/RTSP for Internet streaming is being specified. Usage with other protocols is possible, but has not been explored yet.

The Generic URI Scheme defines two manners in which subparts of Web resources can be addressed: queries and fragments. Queries are given through extending the Web resource's address by a string that is separated through the special character "?". Fragments are given through extending the Web resource's address by a string that is separated through the special character "#". RFC 3986 specifies that URI queries identify a resource and thus a Web server MUST process the query returning a fully defined resource. There is no generically defined structure to URI queries. However, the format of a CGI query string where name-value pairs are separated by the special character "&" is a commonly used format. The Common Gateway Interface, or CGI, is a convention for external gateway programs to interface with information servers such as HTTP servers (see http://hoohoo.ncsa.uiuc.edu/cgi/). When defining a common manner in which temporal subparts of a Web resource can be addressed, it is important to be compatible with this common CGI format. RFC 3986 also specifies that URI fragments identify a secondary resource by reference to a primary resource and that fragments are interpreted client side, i.e. the Web infrastructure consisting of Web servers and Web proxies will generally ignore the "#" extension of URIs. This basically implies that a fragment provides a particular view on a resource and the view is defined through the type of resource and through the client side application. Both, URI query and URI fragment identifiers have traditionally been defined for specific media types or even specific services. This is especially true for URI queries where the query string may consist of name-value pairs that contain the particular fields and field entries of a particular form that resides on a particular server and is processed by a particular server extension. For the temporal addressing that is proposed in this document, it is important that every Web server that interpretes the query parameter reacts uniformly to it. The particular resources which are covered in this manner are the Annodex and CMML media types. This scheme can be adopted for other media types, by including it in the media type registration for that format.

URI query strings are specified after the special character "?". A temporal query parameter defines one or more intervals of time. Time is given with respect to specific time schemes. The default time scheme is "npt" (normal play time), in which a time point is specified in seconds through a floating point number with an arbitrary temporal resolution. The available time schemes and their specifications are described further down in this document. The BNF for a temporal URI query parameter is: All time intervals actually specify start and end times. If no end timespec is given, it defaults to "infinity", which for a file means the end of the file or for a stream the end of the stream. Overlapping time intervals MUST be interpreted by merging the intervals into one. It is not valid to give several temporal URI query parameters in one URI query. They all need to be wrapped into a single specification. A URI with a query for several intervals may be split by the user agent into several queries for a single interval each and then sent to the server in consecutive requests, which in the case of HTTP/1.1 would e.g. be pipelined. This is of particular interest to user clients that can not handle resources that consist of temporally non-consecutive data, e.g. chained Annodex bitstreams. Examples for URIs containing temporal queries are: http://example.com/video.anx?t=npt:15.2 --- video.anx is transferred from 15.2 seconds into video.anx to the end of the file/stream. http://example.com/video.anx?t=15.2/18.7 --- video .anx is transferred from 15.2 seconds into video.anx to 18.7 seconds; the default time scheme "npt" is used. http://example.com/video.anx?t=npt:15.2/18.7,23 --- video.anx is transferred from 15.2s to 18.7s and from 23s to the end of the file/stream. http://example.com/video.anx?t=npt:15.2/18.7,17.4/30.1 -- video.anx is transferred from 15.2 seconds into video.anx to 30.1 seconds. http://example.com/video.anx?t=npt:15.2/18.7&t=23 --- invalid query parameters - MUST create an error message.

URI fragment specifiers are given after the special character crosshatch ("#"). A temporal URI fragment defines a specific temporal view onto a Web resource consisting of one or more intervals of time. Again, time is given with respect to specific time schemes as defined below, "npt" being the default. The BNF for a temporal URI fragment identifier is (reusing the time-intervalspec of the URI query section above): As with temporal URI query parameters, all temporal intervals are specified through start and end times, the default end time being infinity, which may map to the end of a file or stream. Also, several temporal URI fragment identifiers are not allowed. What a user agent does with a temporal URI fragment is up to itself. As a result of the request being sent to the server, the user agent receives the complete resource. If the user agent is a Web browser and the resource is a video, the user agent MAY play back only the specified time section(s). If the user agent is a sound editor and the resource a sound file, the user agent MAY select the disjoint time intervals in the sound file as a first step to applying some filter to them. Examples for URIs with temporal fragment specifications are: http://example.com/video.anx#t=npt:15.2 --- specifies a view on the section between 15.2 seconds into video.anx and the end of the file/stream. http://example.com/video.anx#t=15.2/18.7 --- specifies a view on the section between 15.2s and 18.7s of video.anx, with a default time scheme of "npt". http://example.com/video.anx#t=npt:15.2/18.7,23 --- specifies a view on the section between 15.2s and 18.7s, as well as 23s to the end of the file/stream. http://example.com/video.anx#t=npt:15.2/18.7,17.4/30.1 -- specifies a view on the section between 15.2s and 30.1s of video.anx.

A time scheme is a short identifier for a type of time specification. The following general time schemes are specified in this document. Further time schemes are expected to emerge and should probably be registered through IANA. "npt" Normal Play Time; base of seconds as used in the RTSP standard "smpte" Society of Motion Picture and Television Engineers time codes as specified in the SMPTE time and control code standard "utc" Universal Time Code giving wall-clock time as specified in the ISO 8601 standard. Specification as BNF: These time schemes are specified as follows: NPT time has a second or subsecond resolution. It is specified as H:M:S.h (npt-hhmmss) or S.h (npt-sec), where H=hours, M=minutes, S=second and h=fractions of a second. Negative values are not allowed. Specification as BNF: SMPTE time codes are optimized for frame level accuracy. SMPTE is specified as HH:MM:SS:FF, where HH=hours, MM=minutes, SS=second, FF=frames. The drop-frame algorithms for calculating the exact times can be found in the references SMPTE standard. Negative values are not allowed. "smpte-24=" SMPTE time with a 24 fps basis "smpte-24-drop=" SMPTE time with a 24/1.001 fps basis "smpte-25=" SMPTE time with a 25 fps basis "smpte-30=" SMPTE time with a 30 fps basis "smpte-30-drop=" SMPTE time with a 30/1.001 fps basis "smpte-50=" SMPTE time with a 50 fps basis "smpte-60=" SMPTE time with a 60 fps basis "smpte-60-drop=" SMPTE time with a 60/1.001 fps basis Specification as BNF: UTC time has a second or subsecond resolution. It is given as YYYYMMDDTHHmmss.hhZ, where Y=year, M=month, D=day, H=hour, m=minute, s=second, h=subseconds (one-hundredth of a second). Specification as BNF: utc-hhmmss = 6DIGIT [ "." 1*DIGIT ] ; < HHMMSS.fraction > ]]> Examples for time-interval URI specifications with different time schemes are:

The semantic interpretation of time specifications given with any of the schemes depends on the resource. With every resource there are two associated basetimes: a UTC basetime which may e.g. specify the creation time of the resource, and a playback basetime used for display in a user agent while presenting the resource. The playback basetime of a resource defaults to 0 seconds unless the resource has a different basetime associated with it. A contrasting example: in professional video production, the first frame of video of a program normally refers to a SMPTE basetime of 01:00:00:00, not 00:00:00:00. This practice arose from the requirements of program production and analog videotape recording technology, and it has subsequently become a uniform, almost ironclad practice worldwide. Associating such a practice to a digital video resource requires a way to store that basetime with the resource and interpreting it correctly when addressing via temporal URIs. Annodex provides such a mapping through a field in its headers that stores the playback basetime. CMML has a tag in its stream element for it. Examples: If a resource has an associated basetime of 3600 seconds, and the given temporal fragment offset is 4000 sec, only a seek of 400 sec into the resource is required. If the basetime is given as clock time 20001010T142211.23Z and the temporal offset specified is 20001010T145511.23Z, an offset of 33 minutes into the resource is requested. The UTC basetime of a resource defaults to being non-specified. Associating such a UTC basetime with a resource requires a way to store that basetime with the resource. For example, for a resource that is a file on a server, it may be chosen to be the time of storage of that resource. Annodex also provides a field in its headers to store the UTC basetime, and CMML has a tag for it. The semantic interpretation of temporal intervals is as IN and OUT times like the conventions in use for editing media content: [time_from;time_to[ . This means that the interval stops just at the end time. The reasoning is that it just works when concatenating two intervals that follow directly upon each other.

Time-continuous resources often come with high bandwidth requirements, so a HTTP server SHOULD serve out only the queried time interval of a base resource to avoid unnecessary network load. For example, a 1 hour Digital Video (DV format) requires about 25 GB (MPEG-2 reduces that to about 3 GB). This also significantly reduces the delay for the user agent for receiving relevant data. A user agent that sends a URI that includes a temporal query to a HTTP server expects the server to return just the subparts of the base resource that it is querying for. It expects to be notified by the server if the server cannot resolve the query such that it may take appropriate action. The HTTP protocol specifies that a server MUST return a "404 Not Found" error message on URI requests which cannot be resolved. A URI with a query component is a different URI than the same URI without a query component, so if it cannot be resolved, a "404 Not Found" is expected . However, most current implementations of HTTP servers regard the query component as a relative to the same URI without the query component and therefore support a somewhat non-conformant resolution of such a URI request: if they cannot resolve the query component, they will try to resolve the same URI without the query component. This makes it difficult for a user agent to find out whether a temporal query component was actually resolved or not. To address this issue, a HTTP server that is capable of resolving temporal URI requests on a specified resource MUST return an additional accept header field that indicates that it can handle the temporal URI addressing and for which time schemes it can handle it. The header field is called "X-Accept-TimeURI" and the value is a list of time schemes, e.g. "X-Accept-TimeURI: npt, smpte-24, smpte-25, smpte-30, smpte-50, smpte-60". This request header is included for all URI requests to resources on which the HTTP server can resolve a temporal URI request. A user agent SHOULD check the "X-Accept-TimeURI" field to decide whether its temporal URI request has been honoured and thus the results can be displayed accurately.

Caching HTTP/1.1 proxy servers allow storage of requested Web resources closer to the requesting user agent and reuse by other close-by user agents, reducing bandwidth usage and access time. The HTTP/1.1 caching mechanisms also works for time-based Web resources. However, complete time-based Web resources, such as Annodex resources, are often memory-intensive. User agents may more frequently request URIs with temporal queries than the full resource. Therefore, a larger impact on bandwidth usage is expected from caching the temporal URI queries as they are independent resources. The defined caching mechanism in HTTP/1.1 for subranges of a resource is based on storage of byte subranges. For time-based Web resources for which it is possible to map a temporal URI query to a set of byte subranges on the base resource, URIs with time-queries can also be stored in a proxy's cache. To this end, the HTTP server MUST ensure that the core data that relates to the temporal subpart is bitwise identical to the original data - otherwise a byte range cannot be defined. This is achieved for Annodex subresources by changing only the control section but not the data section on remuxing, as described in the Annodex specifiation. Mapping of temporal URI queries to byte ranges can only happen on the origin server, being the only component that holds the base resource and can thus understand the relationship between time and bytes. The caching mechanism in HTTP/1.1 for byte ranges is based on the user agent requesting a URI with a "Range" request header that specifies the byte ranges. Thus, an additional agent-driven negotiation for delivery of the byte range mapping prior to the "Range" request is introduced to enable support of temporal URI queries. A request header of "X-Accept-Range-Redirect" is required to indicate to the server that a mapping of the temporal URI query parameters to a byte range is requested rather than delivery of the query-part of the resource straight away. As this field initiates the server to provide a different response than if this field was not available, the server MUST include into its response a message header of "Vary: X-Accept-Range-Redirect". It MAY also indicate with "Accept-Ranges: bytes" that it is able to accept byte range requests. And it MAY indicate with "X-Accept-TimeURI" that it has processed the temporal URI query request. The server will return a list of the byte ranges that the user agent should ask for in another additional response header field: "X-Range-Redirect". It will also need to redirect the user agent to another resource, namely the base resource, which it provides in the "Location" header field. Data that is specific to the URI request with the given time-query MUST then be returned to the user agent in the message body, such as for example an adapted control section of an Annodex file. The response is a "200 OK" as the request was satisfied with this response. After this, the user agent has all the information it requires to post another GET request, this time for the base resource only and including a "Range" request header field. In its response, the server includes the requested one or several byte ranges. If several byte ranges are required, the response will be a multipart message as defined in the HTTP/1.1 specification. The response message headers include the "Accept-Ranges: bytes" header, and the "Vary: Range" header, and provides the byte range(s) in the "Content-Range" header. This is an example HTTP message exchange: User Agent HTTP request: Server HTTP response: User Agent HTTP request: Server HTTP response:

The X-Accept-Range-Redirect request header field prompts the server to reply with a byte range specification that maps the queried time ranges of a URI.

The X-Range-Redirect response header field provides a list of byte ranges to which the server redirects a user agent for finding the rest of the data that was requested. An example of this field is:

The X-Accept-TimeURI response header field returns for a timed URI query what time schemes it can resolve and thus implicitly also whether it has resolved it. An example of this field is:

Initial discussions within the AV Transport Working Goup have started about how to use temporal URI addressing over RTP/RTSP. It doesn't seem to make sense to distinguish between fragments and queries as in the streaming scenario everything is focused on being performed on the server. Therefore the idea is not to bother with query specifications and only focus on fragment specifications. When interpreting temporal fragment offsets in RTP/RTSP, the user agent transforms it into a Range specification in the PLAY request header field. Multiple time ranges are easily queued by several PLAY requests. For example, a requested URI of rtsp://foo.bar/nemo#t=npt:10 will e.g. result in the following PLAY request: An implementation of this specification has not been put forward yet. Also note that RealNetworks is using a different syntax for querying temporal intervals of RealMedia over RTP/RTSP.

This specification does not create any new security issues beyond the ones already specified for URIs in RFC 3986.

draft-pfeiffer-temporal-fragments-01: Extension of the number of available SMPTE time-schemes. Many thanks to Bill Miller and Oliver Morgan of the SMPTE for their input on these changes. Deleted "start" and "now" as time specification values. Extension of the temporal fragment addressing to also address temporal intervals, not only time points. Added section that includes some key points of discussion where the existing URI standard contradicts the use of fragments for time-continuous data. draft-pfeiffer-temporal-fragments-02: Full compatibility with existing URI standard specification is achieved by using both, query and fragment specifications for addressing. Restriction of specification to http scheme (Web) only, as usage on other protocols (such as rtp/rtsp) still needs to be explored. All addressing is interval based because an offset really implies a view onto the document from that point onwards. draft-pfeiffer-temporal-fragments-03: Restricted the specification for Annodex and CMML currently with only a suggestion to adopt it for other formats. Changed the specification of periods of time to be compatible with ISO 8601, i.e. replaced "-" with "/". This is also very useful for time specs that contain a "-" like most of the ISO 8601 specs. Thanks to Dean Blackketter for pointing it out. Replaced the word "timebase" with the word "basetime" as that seems to be the more commonly used one to specify a timestamp for the start of a stream of time-sampled data. Added a section on how to use timed URIs with HTTP. This includes caching Web proxies and added HTTP message headers. Started a section on how to use timed URIs with RTP/RTSP. Some initial feedback from the AVT Working Group at IETF seems to indicate not to bother with rtsp queries and just to focus on fragments. This may not be quite conformant to the URI RFC because these RTP/RTSP fragments would be handled on the server - yet this has not been resolved for any use of fragments over RTP/RTSP. draft-pfeiffer-temporal-fragments-04: Fixed some of the examples for the specification of samples with times, which should now include "t=".

Harvard University

29 Oxford Street Cambridge MA 02138 US +1 617 495 3864 sob@harvard.edu

World Wide Web Consortium

Massachusetts Institute of Technology 77 Massachusetts Avenue Cambridge MA 02139 US +1 617 253 5702 +1 617 258 5999 timbl@w3.org

Day Software

5251 California Ave., Suite 110 Irvine CA 92617 US +1 949 679 2960 +1 949 679 2927 fielding@gbiv.com

Adobe Systems Incorporated

345 Park Ave San Jose CA 95110 US +1 408 536 3024 LMM@acm.org

Columbia University

Dept. of Computer Science 1214 Amsterdam Avenue New York NY 10027 US schulzrinne@cs.columbia.edu

Netscape Communications Corp.

501 E. Middlefield Road Mountain View CA 94043 US anup@netscape.com

RealNetworks

1111 Third Avenue Suite 2900 Seattle WA 98101 US robla@real.com

The Society of Motion Picture and Television Engineers

595 W. Hartsdale Ave. White Plains NY 10607 USA smpte@smpte.org

International Organization for Standardization

1 rue de Varembre Case Postale 56 Geneva 20 1211 CH central@iso.org

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4180 Silvia.Pfeiffer@csiro.au http://www.ict.csiro.au/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4222 Conrad.Parker@csiro.au http://www.ict.csiro.au/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4222 Andre.Pang@csiro.au http://www.ict.csiro.au/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4180 Silvia.Pfeiffer@csiro.au http://www.ict.csiro.au/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4222 Conrad.Parker@csiro.au http://www.ict.csiro.au/

Commonwealth Scientific and Industrial Research Organisation CSIRO, Australia

PO Box 76 Epping NSW 1710 Australia +61 2 9372 4222 Andre.Pang@csiro.au http://www.ict.csiro.au/

University of California, Irvine

Department of Information and Computer Science University of California, Irvine Irvine CA 92697-3425 US +1 949 824 7403 +1 949 824 1715 fielding@ics.uci.edu

World Wide Web Consortium

MIT Laboratory for Computer Science 545 Technology Square Cambridge MA 02139 US +1 617 258 8682 jg@w3.org

Western Research Laboratory

Compaq Computer Corporation 250 University Avenue Palo Alto CA 94305 US mogul@wrl.dec.com

World Wide Web Consortium

MIT Laboratory for Computer Science 545 Technology Square Cambridge MA 02139 US +1 617 258 8682 frystyk@w3.org

Xerox Corporation

3333 Coyote Hill Road Palo Alto CA 94034 US +1 650 812 4365 +1 650 812 4333 masinter@parc.xerox.com

Microsoft Corporation

1 Microsoft Way Redmond WA 98052 US paulle@microsoft.com

World Wide Web Consortium

MIT Laboratory for Computer Science 545 Technology Square Cambridge MA 02139 US +1 617 253 5702 +1 617 258 8682 timbl@w3.org

The authors greatly acknowledge the contributions of Rob Collins, Zen Kavanagh, and Andrew Nesbit in developing this specification. We also thank Bill Miller and Oliver Morgan of the SMPTE, Dave Singer from Apple Computer Inc., and Dean Blackketter for their contributions. The many comments on this document from the URI discussion group at the W3C (uri@w3.org), especially the comments from Larry Masinter, Al Gilman and Martin Duerst and others have strongly shaped the current format. Many thanks also to the Audio/Video Transport working group at the IETF (avt@ietf.org), foremost to Magnus Westerlund, for giving valuable feedback on how to improve the specification and picking up the discussion around how to use the scheme on RTP/RTSP. Last but not least thanks go to the ISO/MPEG working group with whom harmonisation in addressing formats is still pursued - thanks to Ian Burnett, Myriam Amielh, Ernest Wan, and Gerrard Drury.