WTVP

WTVP, or the WebTV Protocol, is the protocol that clients from the first generation of WebTV/MSN TV (boxes/Viewer/Dreamcast) use to access the WebTV/MSN TV network and content hosted by it, and is also the protocol that is understood by frontend servers in the WebTV/MSN TV service architecture. WTVP operates on top of TCP and is based on the HTTP 1.0 protocol, adding its own extensions on top of it such as persistent connections, encryption and authentication "using a shared secret"[1], compression, ticket-based authorization, and specialized headers designed to trigger certain commands on WebTV/MSN TV clients. Content delivered by WTVP can be in any format, be it proprietary formats specifically designed for WebTV/MSN TV clients or otherwise. Pages designed to be viewed by the end user, though, are usually served as HTML, with dynamic portions on said pages usually being generated server-side with the HTML or utilizing JavaScript when being loaded on the browser.

Information on this protocol and how it was exactly designed to work is very sparse as of writing. If you have any crucial information on this protocol or anything pertaining to it, then don't hesitate to send it over to us!

Format

Seeing as WTVP is heavily based off of HTTP 1.0, many concepts from that have been carried over to WTVP with barely any differences, such as headers, methods, and status codes. This leaves the general format of the protocol unchanged. The most important things to note, though, are that WTVP messages are still based on a human-readable, line-separated format, messages do not carry a version parameter as usually seen in HTTP messages, and a delimiter is used to separate lines in protocol messages. In messages sent by the client, the delimiter is a CRLF sequence (\r\n), but in server messages, it's believed that it was reduced to a single line feed (\n). Clients will accept server messages using CRLF as the delimiter regardless.

Requests

WTVP requests follow a similar format to HTTP 1.0's requests. They consist of a method to perform the request with, a path identifying the resource being requested, headers, and if appropriate, an entity body. Headers are essential to most WTVP communication and allow frontend servers to tell the client what it should do or what information to store. Entity bodies, however, are for the most part optional in WTVP. Known supported methods in WTVP are GET, POST, HELP, and SECURE. [2] GET and POST requests follow this format:

{METHOD} {PATH}\r\n
[Header1: value\r\n]
[...]
[HeaderN: value\r\n]
\r\n
[BODY]

The path in a WTVP request can either be a service URL (wtv-XXX:/), an absolute path (/xxx/yyy/...), or any standard URI (like "http://www.example.com"). Headers can be a mix of standard HTTP headers and WTVP's proprietary headers, and you can find a list of all known headers used by WTVP on this wiki ("Headers > Header List" on the top-right menu), and additionally, a list of commonly observed headers usually sent in WTVP requests by both the Viewer and boxes. The entity body is usually only present in WTVP server responses if necessary, or POST requests sent to a frontend server. If the entity body is present in a request, then a Content-length header must be present containing the numeric length of the entity body, and a Content-type header can be present as well to identify the type of data being transported.

Getting to how SECURE requests are formatted, those are a bit different from GET and POST requests in that they have expected headers which are almost always sent with the request. A SECURE request will look like the following (headers we believe that are expected are marked in bold in the example):

SECURE ON\r\n
Accept-Language: en-US,en\r\n
wtv-ticket: {wtv-ticket}\r\n
wtv-connect-session-id: 1499ac58\r\n
wtv-client-serial-number: {SSID}\r\n
wtv-encryption: true\r\n
wtv-capability-flags: {capability flags}\r\n
wtv-system-version: 16250\r\n
wtv-client-rom-type: US-BPS-flashdisk-0MB-8MB-softmodem-CPU5230\r\n
wtv-client-bootrom-version: 2525\r\n
wtv-system-chipversion: 84017152\r\n
wtv-system-sysconfig: 3133702\r\n
wtv-system-cpuspeed: 147993946\r\n
wtv-disk-size: 3990\r\n
wtv-script-id: 1611862638\r\n
wtv-script-mod: 1587909919\r\n
wtv-incarnation:47\r\n
\r\n

For SECURE requests, the area where the path would be is now replaced with the string "ON". They also don't carry any data for specific requests alone. SECURE requests simply signal a frontend server to start intercepting encrypted RC4 streams for actual requests, and after being sent once to a connected frontend server, a client will simply continue to send the encrypted form of further data without any preceding request. The encryption method is discussed further in this page under "SECURE method (encryption)."

So far, we do not know how the HELP method works, nor do we have any captures of WTVP requests using this method or any confirmation on if it's a valid method at all.

Responses

WTVP responses are also similar to HTTP 1.0's, consisting of a status code, a string explaining the aforementioned status code ("reason phrase"), headers, and an entity body, the last two being optional. The rules of the entity body requiring Content-length headers and allowing Content-type headers also apply to responses. The status code in the original RFC draft for HTTP 1.0 is expected to be 3 digits, but WTVP's status codes appear to be variable length since 4-digit status codes have been observed in network traffic as well.

{STATUS} {REASON}\n
[Header1: value\n]
[...]
[HeaderN: value\n]
\n
[BODY]

The delimiter used to end lines in official server responses is believed to be a single line feed as opposed to the CRLF combination seen in client messages, and while clients might accept server responses with CRLF delimiters nonetheless, WTVP data snippets on this wiki will reflect the single line feed in server messages for accuracy.

For certain status codes, the WebTV/MSN TV firmware will simply use the reason phrase as the message to show to the end-user.

WTVP responses commonly carry a Connection header with the value Keep-Alive, but the purpose of this header being there exactly is unknown, assuming that WTVP inherently allowed persistent connections by design.

SECURE responses from our research follow the same format as plaintext ones. When transporting encrypted data in the entity body, though, the response has expected headers like with the SECURE requests (expected headers are marked in bold):

{STATUS} {REASON}\n
Connection: Keep-Alive\n
wtv-encrypted: true\n
wtv-lzpf: 0\n
Content-length: {content-length}\n
Content-type: {content-type}\n
[...]
\n
{BODY}

The wtv-lzpf header in these responses is not strictly linked to encryption, and is used to compress response bodies. In this case, the service compresses the actual data before encrypting it and sending it over the wire.

A list of currently known status codes and their reason phrases is linked in the top right menu as "Status Codes".

Command Headers

An interesting use of certain headers by WTVP is having them act as small commands to manipulate the WebTV/MSN TV client, and usually having multiple of them in sequence. These headers will be referred to as "command headers" on this wiki. These are used in situations where the server has to instruct the client to perform a command or a sequence of commands, like reboot, flush its cache, or update its service list. Any headers on the Header List page that we consider to function as a command header will have an appropriate column marked to label it as such.

Service Identification

WTVP services on the server-side are separated by an identifier and individual ports, rather than through 100% physical means, as in different machines are not always expected to handle individual services. This is especially true in the original service architecture as several services often got "pooled" onto a single frontend server to reduce costs. Most services would be identified with a "wtv-" prefix, and a short name to identify what content the service hosts (e.g., "wtv-1800" for registration of new clients, "wtv-head-waiter" for the headwaiter service, "wtv-home" for home page content, etc.). Technically though, they can take on any name, which is especially the case for the proxies used for web browsing by WebTV/MSN TV clients, like with the HTTP proxies using the "http" identifier and FTP proxies using the "ftp" identifier, to name a couple. A service can also have multiple identifiers, like how "wtv-chat" has the alternate identifier of "wtvchat" (note the lack of a hyphen).

The URI for these services on the WebTV/MSN TV network use their identifier followed by a colon and optionally, one or two forward slashes based on our knowledge of how WebTV/MSN TV handled URIs. In the case of internal service content, this allowed service pages to reference said content without the risk of letting anyone easily access it outside the network.

A list of known WTVP services is linked in the top right menu as "List of WTVP Services".

Compression

WTVP can compress responses bodies if needed. The algorithm used to compress responses is known as "LZPF", based on the Huffman compression algorithm. While no proper documentation on this compression algorithm exists on the internet as of now, eMac's ROMFS libraries (GitHub mirror) do contain code for an LZPF library written in Python. The criteria that makes response data eligible for compression isn't entirely clear, although from observing official service responses, it only ever gets applied to HTML or plain text data

It is assumed that a WTVP response is compressed if the wtv-lzpf header is present. The Content-length header won't match the length of the compressed data in this case as well. The data will have to be buffered as soon as it arrives from the network transport, and then has to be uncompressed for the client to determine if the length of the plaintext data it has matches with the one specified in Content-length.

Web Browsing

To allow WebTV/MSN TV clients to access web sites outside the WebTV/MSN TV intranet, a proxy service running on top of WTVP will be defined in the service list for whatever protocols the service supports for proxying, which the client will use to access those external sites. So far it's been recorded that WebTV/MSN TV had proxy services for HTTP, FTP, and Gopher. Not much documentation on how the proxy services worked is available at the moment, though.

Security

Ticket-based authorization (wtv-ticket)

Authorization for WTVP services past the headwaiter involve one global header: the wtv-ticket header. For more information: WTVP/Tickets

SECURE method (encryption)

The SECURE method is to WTVP as HTTPS is to HTTP. It allows both a WebTV/MSN TV client and the servers to communicate with an encryption layer, protecting messages from being intercepted on the wire. Encryption involves RC4, a wtv-incarnation sequence number incremented and sent with initial WTVP messages (both plaintext and SECURE ON messages) on first opening a connection to a frontend server, and two session keys generated for the client's session obtained during the challenge/response process (which are also included in tickets). When the client receives the wtv-challenge on the headwaiter login service, two special 16-byte values are sent along with the other challenge information. These are the two session keys needed to encrypt and decrypt SECURE WTVP messages. Immediately after the client sends the challenge response to the headwaiter do messages start being encrypted, with subsequent responses sent from the headwaiter being encrypted as well, and the client is free to send any SECURE requests to further services that support it.

To encrypt a WTVP message, the wtv-incarnation value and one of the session keys must be used, depending on what message is being encrypted. The first session key from the challenge is used exclusively for entire client -> server messages, and the second one used exclusively for entity body data from server messages, if applicable. wtv-incarnation is packed into a big-endian 32-bit value, and that and the chosen session key are then transformed and hashed with MD5 to create the resulting encryption key, in a manner shown by the following psuedocode:

MD5(WTV_SESSION_KEY + INCARNATION + WTV_SESSION_KEY)

RC4 encryption is then used to encrypt WTVP messages (or just entity body data for server -> client messages), using the MD5 hash generated earlier as a key.

Encrypted messages are treated as a constant stream, where they have to be decrypted as they are sent and then buffered to make sure messages are fully received.