This article describes the most efficient
way to continue downloading a file from a point where it stoped and
organize the mutithreaded downloading feature using the WinInet library.
When you download a file or some other
resource from the web server, sometimes connection errors may occur.
These lead to connections being lost and forces you to reconnect and
start the downloading process again. From here it would be a great idea
if you could continue the downloading of the file from the point of
interruption.
In order to implement this functionality
you need a method which allows you to position the remote resource
pointer at some random place inside the resource. Lets go and consider
the WinInet library and its interfaces which provides users with
the possibility of organizing the downloading process from the point
where it stops (named the resuming feature).
The Microsoft WinInet library provides users with the InternetSetFilePointer function
which sets a file position for the InternetReadFile function. You can
call up InternetSetFilePointer the next time when you are about to
continue downloading in order to move to the last error place. The
sample code below demonstrates how to use this function for the
resuming issue:
var hOpen, hConnect, hResource: HINTERNET; DataProceed: array[0..8191] of Byte; numread: DWORD; begin hOpen := InternetOpen('WinInet resuming sample', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0); hConnect := InternetConnect(hOpen, PChar(host), INTERNET_DEFAULT_HTTP_PORT, nil, nil, INTERNET_SERVICE_HTTP, 0, 0); hResource := HttpOpenRequest(hConnect, 'GET', PChar(resource), nil, nil, nil, 0, 0); HttpSendRequest(hResource, nil, 0, nil, 0);
InternetSetFilePointer(hResource, Position, nil, FILE_BEGIN, 0) releat InternetReadFile(hResource, @DataProceed, SizeOf(DataProceed), numread); ... until (numread <= 0);
InternetCloseHandle(hConnect); InternetCloseHandle(hOpen); end; |
There are some serious restrictions in
using of the InternetSetFilePointer function. Among them the demand
that the WinInet cache should be enabled. As a result you have two
copies of the resource - one of them is in the local file to which you
store the data and the other one within the Internet Temporary Files.
In most cases this is not a problem. So if you are about to implement
the resource resuming feature only, you can freely use the
InternetSetFilePointer function without any problems.
Another thing you will not be able to
peform with this mechanism is: simultaneous resource access from
different threads. This method is used in multithreaded downloading
mode. The multipart multithreaded downloading allows you to improve the
performance, by fully using the TCP channel.
Lets consider it in more detail. Suppose
you are about to start two downloading threads. The first thread starts
the downloading from beginning of the file while the second thread
starts from the middle of the same file. The second thread should set
the file position but the InternetSetFilePointer function is unable to
perform it because the whole of the previous file content is not yet
cached in the WinInet cache.
To work around this problem lets use another approach - range request specifing.
With using the HttpAddRequestHeaders WinInet
function it is possible to set-up the additional request header which
contains the range of data in bytes you are about to retrieve from
server. The sample below demonstrates using of HttpAddRequestHeaders:
procedure TMyThread.SetResourcePos(Resource: HINTERNET; ResourcePos, BytesToProceed: Integer); var s: string; begin s := Format('Range: bytes=%d-%d', [ResourcePos, ResourcePos + BytesToProceed]); HttpAddRequestHeaders(Resource, PChar(s), Length(s), HTTP_ADDREQ_FLAG_ADD_IF_NEW); end; |
The sample code which allows to start the downloading with random data range is shown below:
var hOpen, hConnect, hResource: HINTERNET; s: string; DataProceed: array[0..8191] of Byte; numread: DWORD; begin hOpen := InternetOpen('WinInet resuming sample', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0); hConnect := InternetConnect(hOpen, PChar(host), INTERNET_DEFAULT_HTTP_PORT, nil, nil, INTERNET_SERVICE_HTTP, 0, 0); hResource := HttpOpenRequest(hConnect, 'GET', PChar(resource), nil, nil, nil, 0, 0);
s := Format('Range: bytes=%d-', [Position]); HttpAddRequestHeaders(hResource, PChar(s), Length(s), HTTP_ADDREQ_FLAG_ADD_IF_NEW); HttpSendRequest(hResource, nil, 0, nil, 0); repeat ZeroMemory(@DataProceed, SizeOf(DataProceed)); InternetReadFile(hResource, @DataProceed, SizeOf(DataProceed), numread); ... until (numread <= 0);
InternetCloseHandle(hConnect); InternetCloseHandle(hOpen); end; |
As you can see an example above works much faster than example with the InternetSetFilePointer.
All the algorithms we considered in this
paper are operable only when the web site from which the downloadable
file belongs supports the random resource access. If not, the "range"
request being sent to the web server will be ignored and the whole
amount of data will be sent to the client. Before using the range
request header or an algorithm with InternetSetFilePointer call, please
check whether the web server being connected to supports the random
access feature.
In order to perform this check in the easiest way you can use the following code:
AllowsRandomAccess := Integer(InternetSetFilePointer(hResource, 0, nil, FILE_END, 0)) > 0; |
Also please note that
InternetSetFilePointer cannot be used reliably if the content length is
unknown. So you should also perform this check as follows:
var reslen: Integer; buflen, tmp: DWORD; begin buflen := SizeOf(reslen); tmp := 0; reslen = 0; HttpQueryInfo(hResource, HTTP_QUERY_CONTENT_LENGTH or HTTP_QUERY_FLAG_NUMBER, @reslen, buflen, tmp); AllowsRandomAccess := (reslen > 0); end; |
Retrieving of the Content-Lenght resouce
information is very important for range requests too when they are used
for multipart multithreaded downloading mode. If you are simply
organizing the resuming of the download after an error has occured you
can simple specify the first range value and continue to retrieve the
data from error point up to the end.
The full source code of samples being used in this article can be downloaded at:
Please note that
WinInet error handling was not implemented in these examples in order
to keep them as simple as possible. No warning messages will appear in
case of invalid URL. You can easily implement error handling if such
issue is important to you.
While writing this article we used the following resources:
- http://msdn.microsoft.com - Microsoft MSDN Library
- http://www.w3.org/Protocols/rfc2616/rfc2616.html - Hypertext Transfer Protocol -- HTTP/1.1 (RFC 2616)
This code is constantly being improved and your comments and suggestions are always welcome.
Please write us at info@clevercomponents.com
http://www.clevercomponents.com/articles/article015/resuming.asp