Error: Content Length mismatch HTTP header

[xismn]

Hello,

This is a long question, and it doesn't have anything to do with C++. I've decided to post this in the lounge since the other boards aren't appropriate, and I didn't feel like finding a more niche forum. I know this forum is pretty busy and there are a lot of folks here that specialize in several domains.

I have to preface my question by saying that, recently, I've been trying to download some .mp4's being streamed by a JWPlayer on some website. This is pretty easy to do - in Chrome's dev console you can log requests to media resources and find the direct URL to a resource and simply download it from there - which is what I've been doing with much success.

There is, however, one video which I cannot access. The JWPlayer will display the video with no problems, but when I attempt to access the video resource directly Chrome returns an error, namely: ERR_CONTENT_LENGTH_MISTMATCH. I'm aware that this error is a result of the request and response headers' Content-Length attributes not matching up (The server is misconfigured, and reports a Content-Length that is completely divorced from reality - i.e. the file sizes differ).

I also noticed that my version of Internet Explorer will open the video resource in Windows Media Player by default. It won't try to download it, or even let me. The video will stream in Windows Media Player with out any problems, but I still won't be able to download it.

I've tried several different options to circumvent this problem. I've tried downloading and installing different versions of different browsers which apparently "ignore" the Content-Length attribute of these HTTP headers and will display resources with mismatched lengths according to this archived blog post:

http://blogs.msdn.com/b/ieinternals/archive/2011/03/09/browsers-accommodate-incorrect-http-content-length-and-sites-depressingly-depend-on-it.aspx

Which didn't work.

I've also tried capturing the stream in VLC player - didn't work.
I've tried tampering with my default applications, to see if I could force certain browsers to stream and others to download - didn't work.

I wrote a program in C++ using libCurl to try and download the resource. I set libCurl to be as verbose as possible, so here are the details from one request to access the resource:

HTTP/1.1 200 OK
Date: REDACTED
Server: REDACTED
X-Powered-By: PHP/5.4.45
Set-Cookie: REDACTED
Expires: REDACTED
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Accept-Ranges: bytes
Content-Length: 273628708
Content-Type: video/mp4
X-Varnish: 33423606
Age: 0
Via: 1.1 varnish-v4
Connection: keep-alive
Accept-Ranges: bytes

transfer closed with 273628708 bytes remaining to read

Note the error. The connection was terminated, and not even a single byte of data had been transferred. I then proceeded to enable different options such as changing CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_1_0 (I read somewhere that certain versions of the protocol handle these header attributes differently, or ignore them entirely). I also tried to setting chunked Transfer-Encoding, as well as setting CURLOPT_IGNORE_CONTENT_LENGTH, only to receive an error stating "no chunk, no close, no size. Assume close to signal end". I've tried several different permutations of these configurations. None of this worked.

I'm pretty desperate at this point. Obviously, the server hosting the video is misconfigured - and it's just for this particular video. Other videos from the same server can be accessed and downloaded correctly.

I guess my question is, does anyone here have any experience with this error? Any other suggestions? Also, how are Windows Media Player and the JWPlayer able to stream this apparently corrupted resource? Thanks for any information.

EDIT - As you can see, the video I'm trying to download is 273628708 bytes in size, approximately 1.5 hours of footage. Could this have something to do with it?

[tition]

Can you post your C++ libcurl code - at least the main portion/loop of it?

[JLBorges]

Since the server has sent this:

Accept-Ranges: bytes
Content-Length: 273628708

Try to download the file in parts by making range requests.
http://curl.haxx.se/libcurl/c/CURLOPT_RANGE.html

[xismn]

@tition Here's the whole program, just for completeness:

#include <iostream>
#include <string>
#include <sstream>
#include <fstream>
#include <curl/curl.h>

#ifdef _DEBUG
#pragma comment(lib, "curl/debug/curllib.lib")
#else
#pragma comment(lib, "curl/release/curllib.lib")
#endif


std::size_t write_data(char* ptr, std::size_t size, std::size_t nmemb, void* userdata) {
	std::ostringstream* stream = (std::ostringstream*)userdata;
	std::size_t count = size * nmemb;
	stream->write(ptr, count);
	return count;
}

int main(int argc, char* argv[]) {

	curl_global_init(CURL_GLOBAL_ALL);
	CURL* handle = curl_easy_init();
	CURLcode result;
	std::ostringstream stream;
	std::ofstream file_dump("test.mp4", std::ofstream::out | std::ofstream::binary);

	std::string string_error = "";
	const std::string string_url_base = "";//redacted
	char errbuffer[CURL_ERROR_SIZE] = {};

	//curl_easy_setopt(handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);

	curl_easy_setopt(handle, CURLOPT_ERRORBUFFER, errbuffer);

	curl_easy_setopt(handle, CURLOPT_SSL_VERIFYPEER, 0);
	curl_easy_setopt(handle, CURLOPT_SSL_VERIFYHOST, 0);

	curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, write_data);
	curl_easy_setopt(handle, CURLOPT_WRITEDATA, &stream);
	//curl_easy_setopt(handle, CURLOPT_POSTFIELDSIZE, 0);
	curl_easy_setopt(handle, CURLOPT_VERBOSE, 1L);

	curl_easy_setopt(handle, CURLOPT_URL, string_url_base.c_str());

	curl_easy_setopt(handle, CURLOPT_IGNORE_CONTENT_LENGTH, 1L);

	//struct curl_slist* chunk = NULL;
	//chunk = curl_slist_append(chunk, "Expect:");
	//curl_easy_setopt(handle, CURLOPT_HTTPHEADER, chunk);

	//struct curl_slist* chunk = NULL;
	//chunk = curl_slist_append(chunk, "Transfer-Encoding: chunked");
	//curl_easy_setopt(handle, CURLOPT_HTTPHEADER, chunk);

	result = curl_easy_perform(handle);

	//curl_slist_free_all(chunk);

	string_error = errbuffer;

	file_dump << stream.str();
	curl_easy_cleanup(handle);

	std::cout << string_error << std::endl;

	std::cout << "Finished." << std::endl;
	std::cin.get();
	curl_global_cleanup();
	return 0;
}

Ignore the comments and lack of error-checking. It's just something to get it up and running.

@JLBorges thanks for the interesting link. I tried this with my code, but the server is not cooperating. If I try requesting ranges the server simply gives me a 416 error:

HTTP/1.1 416 Requested Range Not Satisfiable
...
Accept-Ranges: bytes
Content-Range: bytes */0

I did some digging, and I'm under the impression that this has to do with the fact that the size of the resource isn't known beforehand (which, apparently, is indicated by the * in the Content-Range response attribute). I also understand that the HTTP response code I should be hoping for is a 206 partial content response - anyways, is this why the server is rejecting my range requests? Because the size isn't known? Any help is appreciated.

EDIT - When copying the response header from Google Chrome (simply trying to access the resource), it tells me:

Accept-Ranges: 0-273628708
Content-Range: bytes 0-273628707/273628708
Content-Length: 273628708
Content-Type: video/mp4
X-Varnish: 120522620
Age: 0
Via: 1.1 varnish-v4
Connection: keep-alive

Does

Accept-Ranges: 0-273628708

mean that 0-273628708 (i.e. the whole file) is the only acceptable range, or does it mean that any range within that range is apparently acceptable? Not that it matters much because I tried asking for 200, 400, 273628708 bytes etc and I'm still getting a 416 error.