Download file with callback

BlitzMax Forums/BlitzMax Programming/Download file with callback

JoshK

(Posted 2006) [#1]

The command CopyFile() can download a file from the internet and copy it to the user's hard drive. I want to write a new copyfile function with a callback, so that the download progress can be displayed:

Import leadwerks.lwmath

Function CopyFile2:Int(src$,dst$="",callback:Byte Ptr=Null)
	src$=Replace(src,"\","/")
	src=Replace(src,"://","::")
	If dst="" dst=AppDir$+"/"+StripDir(src)
	
	If callback=Null
		Return CopyFile(src,dst)
	Else
	
		Local DownloadProgressCallback(p#)
		DownloadProgressCallback=callback
	
		out:TStream=WriteFile(dst)
		If Not out Return
		in:TStream=ReadFile(src)
		If Not in
			out.close()
			Return
		EndIf
		
		size=in.size()
		While Not Eof(in)
			WriteByte out,ReadByte(in)

			newprogress#=Float(in.pos())/Float(in.size())
			'If Round(newprogress*100.0)>Round(progress#*100.0)
			'	progress=newprogress
			DownloadProgressCallback newprogress
			'EndIf
		Wend
		
		in.close()
		out.close()
		Return True
		
	EndIf
EndFunction

win:TGadget=CreateWindow("",200,200,400,300,Null,WINDOW_TITLEBAR)
Global progbar:TGadget=CreateProgBar(10,10,win.clientwidth()-20,28,win)
CopyFile2("http://www.leadwerks.com/index.php","",DLCallBack)

Function DLCallBack(progress:Float)
	UpdateProgBar progbar,progress
	If PeekEvent() WaitEvent()
EndFunction

The problem is that a URL opened with ReadStream() has an unknown size. If I pre-cache the file with ReadFile(), the entire file is loaded, so I know the size, but I can't display the progress until after the actual file data is already downloaded.

How can I find out the size of a file on the internet, or a stream opened with ReadStream()?

Kev	(Posted 2006) [#2]

a post by Yan sometime back, this will return the files size without downloading.

Print HTTPFileSize("http://YOUR URL TO FILE") + " Bytes"

End


Function HTTPFileSize(url$)
	url$ = url$.Replace("http://", "")
	Local slashPos = url$.Find("/"), host$, file$
	
	If slashPos <> -1
		host$ = url$[..slashPos]
		file$ = url$[slashPos..]
	Else
		Return -1
	EndIf
	
	Local stream:TStream=OpenStream("tcp::" + host$, 80)
	If Not stream Then Return -1	
	
	stream.WriteLine "HEAD " + file$ + " HTTP/1.0"
	stream.WriteLine "Host: " + host$
	stream.WriteLine ""
	
	While Not Eof(stream)
		Local in$ = stream.ReadLine()

		If in$.Find("Content-Length:") <> -1
			stream.Close()
			Return Int(in$[in$.Find(":") + 1..].Trim())
		EndIf	
	Wend
	
	stream.Close()
	Return -1		
End Function

JoshK

(Posted 2006) [#3]

Okay, this works. Now I just need to figure out how to stop the transfer if the connection is broken:

Global win:TGadget=CreateWindow("",Desktop().ClientWidth()/2-200,Desktop().ClientHeight()/2-50,400,100,Null,WINDOW_TITLEBAR)
CreateLabel("Downloading...",10,10,100,16,win)
Global progbar:TGadget=CreateProgBar(10,10+16,win.clientwidth()-20,28,win)
file$="http://blitzbasic.com/file/get.php?file=/Products/demos/Blitz3DDemo183.exe"
If CopyFile2(file$,"",DLCallBack)
	Notify "Download successful."
	OpenURL AppDir$+"\"+StripDir(file)
Else
	Notify "Download failed."
EndIf

Function CopyFile2:Int(src$,dst$="",callback:Byte Ptr=Null)
	url$=src
	src$=Replace(src,"\","/")
	src=Replace(src,"://","::")
	If dst="" dst=AppDir$+"/"+StripDir(src)
	
	If callback=Null
	
		'Just use the regular CopyFile()
		Return CopyFile(src,dst)

	Else
	
		'Set up the callback function
		Local DownloadFileUpdateCallback(p#)
		DownloadFileUpdateCallback=callback

		'Get the file size
		size=FileSize2(url)
		If size=-1 Return
	
		'Open streams
		out:TStream=WriteFile(dst)
		If Not out Return
		in:TStream=ReadStream(src)
		If Not in
			out.close()
			Return
		EndIf
		
		'Copy file and display progress		
		While Not Eof(in)
			pos=pos+1
			WriteByte out,ReadByte(in)
			newprogress#=Float(pos)/Float(size)
			If Round(newprogress*100.0)>Round(progress#*100.0)
				progress=newprogress
				DownloadFileUpdateCallback progress
			EndIf
		Wend
		
		'Close the streams
		in.close()
		out.close()
		
		'Verify file size
		If FileSize(dst)<>size
			DeleteFile dst
			Return
		EndIf
		
		Return True
		
	EndIf
EndFunction

Function Round(val#)
	Local dec#
	dec#=val-Floor(val)
	If dec<0.5 Return Floor(val) Else Return Ceil(val)
EndFunction

Function DLCallBack(progress:Float)
	UpdateProgBar progbar,progress
	If PeekEvent() WaitEvent()
EndFunction

Function FileSize2:Int(url$)
	If Instr(url,"http://")
		url$ = url$.Replace("http://", "")
		Local slashPos = url$.Find("/"), host$, file$
		
		If slashPos <> -1
			host$ = url$[..slashPos]
			file$ = url$[slashPos..]
		Else
			Return -1
		EndIf
		
		Local stream:TStream=OpenStream("tcp::" + host$, 80)
		If Not stream Then Return -1	
		
		stream.WriteLine "HEAD " + file$ + " HTTP/1.0"
		stream.WriteLine "Host: " + host$
		stream.WriteLine ""
		
		While Not Eof(stream)
			Local in$ = stream.ReadLine()
	
			If in$.Find("Content-Length:") <> -1
				stream.Close()
				Return Int(in$[in$.Find(":") + 1..].Trim())
			EndIf	
		Wend
		
		stream.Close()
		Return -1
	Else
		Return FileSize(url)
	EndIf
End Function

JoshK

(Posted 2006) [#4]

Now this is cool. I fixed it so if you the connection is list, the function just returns false.

Global win:TGadget=CreateWindow("",Desktop().ClientWidth()/2-200,Desktop().ClientHeight()/2-50,400,100,Null,WINDOW_TITLEBAR)
CreateLabel("Downloading...",10,10,100,16,win)
Global progbar:TGadget=CreateProgBar(10,10+16,win.clientwidth()-20,28,win)

file$="http://blitzbasic.com/file/get.php?file=/Products/demos/Blitz3DDemo183.exe"

If CopyFile2(file$,"",DisplayDownloadProgress)
	Notify "Download successful."
	OpenURL AppDir$+"\"+StripDir(file)
Else
	Notify "Download failed."
EndIf


Function CopyFile2:Int(src$,dst$="",callback:Byte Ptr=Null,bufsize=128)
	url$=src
	src$=Replace(src,"\","/")
	src=Replace(src,"://","::")
	If dst="" dst=AppDir$+"/"+StripDir(src)
	
	If callback=Null
	
		'Just use the regular CopyFile()
		Return CopyFile(src,dst)

	Else
	
		'Set up the callback function
		Local DownloadFileUpdateCallback(p#)
		DownloadFileUpdateCallback=callback

		'Get the file size
		size=FileSize2(url)
		If size=-1 Return
	
		'Open streams
		out:TStream=WriteFile(dst)
		If Not out Return
		in:TStream=ReadStream(src)
		If Not in
			out.close()
			Return
		EndIf
		
		'Copy file and display progress
		Try
			Local buf:Byte[bufSize]
			While Not in.Eof()
				out.WriteBytes buf,in.Read(buf,bufSize)
				pos=pos+bufsize
				progress#=Float(pos)/Float(size)
				DownloadFileUpdateCallback progress
			Wend
			Catch ex:TStreamWriteException
		EndTry

		'Close the streams
		in.close()
		out.close()
		
		'Verify file size
		If FileSize(dst)<>size
			DeleteFile dst
			Return
		EndIf
		
		Return True
		
	EndIf
EndFunction

Function Round(val#)
	Local dec#
	dec#=val-Floor(val)
	If dec<0.5 Return Floor(val) Else Return Ceil(val)
EndFunction

Function DisplayDownloadProgress(progress:Float)
	UpdateProgBar progbar,progress
	If PeekEvent() WaitEvent()
EndFunction

Function FileSize2:Int(url$)
	If Instr(url,"http://")
		url$ = url$.Replace("http://", "")
		Local slashPos = url$.Find("/"), host$, file$
		
		If slashPos <> -1
			host$ = url$[..slashPos]
			file$ = url$[slashPos..]
		Else
			Return -1
		EndIf
		
		Local stream:TStream=OpenStream("tcp::" + host$, 80)
		If Not stream Then Return -1	
		
		stream.WriteLine "HEAD " + file$ + " HTTP/1.0"
		stream.WriteLine "Host: " + host$
		stream.WriteLine ""
		
		While Not Eof(stream)
			Local in$ = stream.ReadLine()
	
			If in$.Find("Content-Length:") <> -1
				stream.Close()
				Return Int(in$[in$.Find(":") + 1..].Trim())
			EndIf	
		Wend
		
		stream.Close()
		Return -1
	Else
		Return FileSize(url)
	EndIf
End Function

And this will automatically detect and download app updates:

Framework BRL.System
Import Pub.Win32
Import BRL.HTTPStream
Import BRL.Retro
Import BRL.Win32MaxGUI

Extern "win32"
	Function FreeLibrary:Int(hlib)="FreeLibrary@4"
EndExtern

Global SERVERNAME$="www.frauleinspace.com"
Global PRODUCTNAME$="LW3D"
Global PROGBAR:TGADGET
Global BUTTON_CANCEL:TGadget

Global VERSION_MAJOR=0
Global VERSION_MINOR=0
Global VERSION_BUGFIX=0

Notify webupdate()

Function SetUpdateServer(hservername:Byte Ptr)
	servername$=string.fromcstring(hservername)
	MemFree hservername
EndFunction

Function SetUpdateProduct(hproductname:Byte Ptr)
	productname$=string.fromcstring(hproductname)
	MemFree hproductname
EndFunction

Function SetUpdateVersion(major,minor,bugfix)
	VERSION_MAJOR=major
	VERSION_MINOR=minor
	VERSION_BUGFIX=bugfix
EndFunction

Function WebUpdate:Int()

	If Not DeleteFile(CurrentDir$()+"\update.exe") Return
	result=1
	
	Local InternetAttemptConnect:Int(dwReserved:Int)
	hlib=loadlibrarya("wininet.dll")
	If hlib
		InternetAttemptConnect=GetProcAddress(hlib,"InternetAttemptConnect")
		InternetAttemptConnect(0)
		freelibrary hlib
	EndIf
	
	If filesize2("http://"+SERVERNAME+"/data/"+PRODUCTNAME+"/data.ini")=-1 Return
	stream:TStream=ReadFile("http::"+SERVERNAME+"/data/"+PRODUCTNAME+"/data.ini")
	If Not stream Return
	While Not stream.eof()
		s$=stream.readline()
		name$=Lower(Trim(piece(s,1,"=")))
		value$=piece(s,2,"=")
		value=Replace(value,Chr(34),"")
		value=value.trim()
		Select name
			Case "latestversion"
				major=Int(piece(value,1,"."))
				minor=Int(piece(value,2,"."))
				bugfix=Int(piece(value,3,"."))
				If major>VERSION_MAJOR
					update=True
				ElseIf major=VERSION_MAJOR
					If minor>VERSION_MINOR
						update=True
					ElseIf minor=VERSION_MINOR
						If bugfix>VERSION_BUGFIX
							update=True
						Else
							update=False
						EndIf
					Else
						update=False
					EndIf
				Else
					update=False
				EndIf
			Case "latestversionurl"
				updateurl$=value
		EndSelect
	Wend

	If update
		result=result+2
		
		win:TGadget=CreateWindow("",Desktop().ClientWidth()/2-160,Desktop().ClientHeight()/2-50,320,86,Null,WINDOW_HIDDEN)
		hwnd=QueryGadget(win,QUERY_HWND_CLIENT)
		style=getwindowlonga(hwnd,GWL_STYLE)
		setwindowlonga hwnd,GWL_STYLE,style+WS_DLGFRAME
		SetWindowPos hwnd, -1, 0, 0, 0, 0, 3
		SetGadgetShape(win,GadgetX(win),GadgetY(win),GadgetWidth(win)+1,GadgetHeight(win)+1)
		SetGadgetShape(win,GadgetX(win),GadgetY(win),GadgetWidth(win)-1,GadgetHeight(win)-1)
		CreateLabel("Updating...",10,10,100,16,win)
		progbar=CreateProgBar(10,10+16,win.clientwidth()-20,20,win)
		BUTTON_CANCEL=CreateButton("Cancel",win.clientwidth()/2-30,win.clientheight()-26,60,24,win)
		ShowGadget win
		
		If CopyFile2(updateurl$,CurrentDir$()+"\update.exe",DisplayDownloadProgress,64)
			result=result+4
		Else
			
			DeleteFile(CurrentDir$()+"\update.exe")
		EndIf
		
		HideGadget win
		FreeGadget win
		
	EndIf

	Return result
EndFunction

Function DisplayDownloadProgress:Int(progress:Float)
	UpdateProgBar progbar,progress
	If PeekEvent()
		Select WaitEvent()
			Case EVENT_GADGETACTION
				Select EventSource()
					Case BUTTON_CANCEL
						If Confirm("Cancel update?") Return True
				EndSelect
		EndSelect
	EndIf
EndFunction

Function CopyFile2:Int(src$,dst$="",callback:Byte Ptr=Null,bufsize=4096)
	url$=src
	src$=Replace(src,"\","/")
	src=Replace(src,"://","::")
	If dst="" dst=AppDir$+"/"+StripDir(src)
	
	If callback=Null
	
		'Just use the regular CopyFile()
		Return CopyFile(src,dst)

	Else

		'Set up the callback function
		Local DownloadFileUpdateCallback:Int(p#)
		DownloadFileUpdateCallback=callback

		'Get the file size
		size=FileSize2(url)
		If size=-1 Return
	
		'Open streams
		out:TStream=WriteFile(dst)
		If Not out Return
		in:TStream=ReadStream(src)
		If Not in
			out.close()
			Return
		EndIf
		
		'Copy file and display progress
		Try
			Local buf:Byte[bufSize]
			While Not in.Eof()
				out.WriteBytes buf,in.Read(buf,bufSize)
				pos=pos+bufsize
				progress#=Float(pos)/Float(size)
				If DownloadFileUpdateCallback(progress)
					in.close()
					out.close()
					Return False
				EndIf
			Wend
			Catch ex:TStreamWriteException
		EndTry

		'Close the streams
		in.close()
		out.close()
		
		'Verify file size
		If FileSize(dst)<>size
			DeleteFile dst
			Return
		EndIf
		
		Return True
		
	EndIf
EndFunction

Function Piece$(s$,entry,char$=" ")
	Local n
	Local p
	Local a$
	While Instr(s,char+char)
		s=Replace(s,char+char,char)
	Wend
	For n=1 To entry-1
		p=Instr(s,char)
		s=Right(s,Len(s)-p)
		If Not p
			If entry=n Exit Else Return
		EndIf
	Next
	p=Instr(s,char)
	If p<1
		a$=s
	Else
		a=Left(s,p-1)
	EndIf
	Return a
End Function

Function FileSize2:Int(url$)
	If Instr(url,"http://")
		url$ = url$.Replace("http://", "")
		Local slashPos = url$.Find("/"), host$, file$
		
		If slashPos <> -1
			host$ = url$[..slashPos]
			file$ = url$[slashPos..]
		Else
			Return -1
		EndIf
		
		Local stream:TStream=OpenStream("tcp::" + host$, 80)
		If Not stream Then Return -1	
		
		stream.WriteLine "HEAD " + file$ + " HTTP/1.0"
		stream.WriteLine "Host: " + host$
		stream.WriteLine ""
		
		While Not Eof(stream)
			Local in$ = stream.ReadLine()
	
			If in$.Find("Content-Length:") <> -1
				stream.Close()
				Return Int(in$[in$.Find(":") + 1..].Trim())
			EndIf	
		Wend
		
		stream.Close()
		Return -1
	Else
		Return FileSize(url)
	EndIf
End Function