NewsForumsCodeLogsGallerySpecsSearch

Bank/Peek/Poke

Blitz3D Forums/Blitz3D Beginners Area/Bank/Peek/Poke

_PJ_(Posted 2016) [#1]
The fllowing example highlights the issue better than Ican explain it.
I am uncertain how to ensure the CORRECT SIZE of the new 'compressed' bank.


Local n=128

Local TestBank=CreateBank(n)

;Fill testexample
For X=0 To n-1
	PokeByte TestBank,x,Rand(0,7)
Next

; This should resutl in a new bank 3:8 size ratio - but for some reason, the size Max*0.375 is not correct.
Local Shrink=ShrinkBank(TestBank)


Function ShrinkBank(BigBank)
	Local Max=BankSize(BigBank)
	Local ShrinkSize=Max*0.375
	Local Bank=CreateBank(ShrinkSize);Shrink to 3 bits instead of 8 bits
	
	Local BigBytes
	Local BigByteBits
	Local IterBigByte
	Local Word
	Local WordBits
	
	Local SmallByte
	Local SmallShort
	
	Local SmallOffset
	
	For BigBytes =0 To Max-1 Step 7
		DebugLog("Compressing bytes "+Str(BigBytes)+" - "+Str(BigBytes+7)+" of "+Str(Max-1))
		
		Word=0
		For IterBigByte=0 To 7
			BigByteBits=(PeekByte(BigBank,BigBytes+IterBigByte) And 7)
			WordBits=BigByteBits Shl (IterBigByte*3)
			Word=Word+WordBits
		Next
		
		SmallByte=Word And 255
		SmallShort=(Word Shr 8)And 65535
		
		PokeByte Bank,SmallOffset,SmallByte
		PokeShort Bank,SmallOffset+1,SmallShort
		
		DebugLog("Compressed to new Bytes "+Str(SmallOffset)+" - "+Str(SmallOffset+2)+" of "+Str(ShrinkSize-1))
		
		SmallOffset=SmallOffset+3
	Next
	
	Return Bank
End Function



_PJ_(Posted 2016) [#2]
Ignore this.

I found my mistake.

Such a silly thing, too - 

For BigBytes =0 To Max-1 Step 7

Should read
For BigBytes =0 To Max-1 Step 8


______________________________________________________________

Whilst this is quite specific as it stands, the actual concept:

Compressing A*8 bytes where only the first B bits of each byte is used into BA/8 bytes

May be of benefit for others.


virtlands(Posted 2016) [#3]
[obsolete post]


_PJ_(Posted 2016) [#4]
I never once suggested whatsoever this was a general case. It is, as I mentioned clearly "this is quite specific as it stands" for when A=8n and B=3

Obviously it will not work when the compressed bits overflow an 8bit byte.

Perhaps I should have clarified that the values MUST be integral. This includes the BA/8 compressed size.

Yes it could be modified to include appended "padding" bits as well as cases where different (but consistent per function call) numbers of relevant bits are used to be compressed to related sizes of bytes.

I have neither care nor interest in doing so, but you're welcome to develop further should you wish.

This was not my intention, and is of no interest nor use to me. This was code for a specific purpose in which a bank of size in which only the first 3 bits were relevant and therefore the information content could be preserved losslessly into a bank of ratio 3:8 the size. That was all. I only posted the function due to the bug mentioned and subsequently addressed in my second post.
On identifying the bug, I deemed it may be interesting to some to see the function.

I am puzzled as to why you would think it would work with a value of A=1.25 knowing that 3.75 is NOT an integer nor a valid size for a bank. or even why you would NOT include the fix given the very reason for posting in the first place was because without the fix mentioned, the vfunction failed.

__
"[ I'll be back later to take a closer look at your project. ] "
What on earth is this supposed to mean? What project, and who the helldo you think you are? Do I presume to oversee or inspect every effort of coding you make? What makes you think I have need nor desirte for your scrutiny of anything I work on? Where exactly do you hope to gain knowledge of my projects or work from?


virtlands(Posted 2016) [#5]
[obsolete post]


RemiD(Posted 2016) [#6]
relax guys, we may disagree on some things and sometimes post critics of others codes/creations, but there is no need to be angry, we are all here to learn and become better coders/developers...


_PJ_(Posted 2016) [#7]
I have no issue with criticising my code, I have no real formal education in coding and so must expect significant areas in which improvements can be made and welcome constructive suggestions. This is why I frequently post here with problems.

I do not, however, welcome other users self-appointing their right to consider themselves arbiters and auditors of work and employ some form of inspection scrutiny whether requested or otherwise with such ridiculous and outright rude commentary as "I'll be back later to take a closer look at your project".

___



The code above is itself a further development (albeit used for a different purpose for a separate project) along a similar notion of lossless compression from an earlier concept relating to compression specific text.
Despite the specifics of text and limtiations on available characters, the original concept did allow a little more versatility and generality of use. I have reproduced a sample blow which contains an example and some information on how it can be adapated for more general usage.

; This can be changed to 16 for SHORT Int or 32 for LONG Int, if the string length is known, this may speed up the process by dealing with more characters at one time. In which case, Remember to change Read/Write Byte to Read/Write Short or Int as required below.

Const BYTESIZE=8 
 
 ;Identifies the distinct characters
 Global CHARSET$="ABCDEFGHIJKLMNOPRSTUVWXYZ. ,'!?" 
 
 
 ;Identifies the number of distinct characters ; (saves time to store in one var rather than call Len() each time
 Global POSSIBLECHARS=Len(CHARSET) 
 
 
 ;This calculates the maximum number of BITS required to represent any of the available Chars. Essentially this is the limit of compression. 
 Global BITS_PER_CHAR=Log(POSSIBLECHARS)/Log(2) ; Yet again, this saves time to calculate and sotr beforehand, rather than call the logarithm function each time.
 
 
 ;Example:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 
 
 
 Local FileOut=WriteFile("TestFile.dat") 
 SetStringToBitStream(FileOut,"THIS IS A TEST! OK, YOU'RE SURE?") 
 CloseFile FileOut 
 Local FileIn=ReadFile("TestFile.dat") 
 Local Name$=GetStringFromBitStream(FileIn) 
 CloseFile FileIn 
 WaitKey() 
 End 
 
 
 
 
 Function GetStringFromBitStream$(Stream) 
 	Local NAME_LENGTH=ReadByte(Stream) 
 	 
 	Local Byte 
 	Local Iter 
 	 
 	Local CharacterBit 
 	Local Bit 
 	Local ByteBit 
 	Local ByteBitValue 
 	Local CharBitValue 
 	Local CurrentChar 
 	Local Char$ 
 	Local Name$ 
 	 
 	Local MaximumByteBits=((BITS_PER_CHAR*NAME_LENGTH)-BYTESIZE) 
 	 
 	Byte=ReadByte(Stream) 
 	 
 	For Bit=0 To BITS_PER_CHAR*NAME_LENGTH 
 		 
 		ByteBit=Bit Mod BYTESIZE 
 		CharacterBit=Bit Mod BITS_PER_CHAR 
 		 
 		ByteBitValue=(2^ByteBit) Mod (2^BYTESIZE) 
 		CharBitValue=(2^CharacterBit) Mod NAME_LENGTH 
 		 
 		If (Byte And ByteBitValue) 
 			CurrentChar=CurrentChar+CharBitValue 
 		End If 
 		 
 		If (CharacterBit=BITS_PER_CHAR-1) 
 			 
 			Char=Mid(CHARSET,CurrentChar+1,1) 
 			 
 			CurrentChar=0 
 			Name=Name+Char 
 		End If 
 		 
 		If ((ByteBit=BYTESIZE-1) And (Bit<MaximumByteBits)) 
 			Byte=ReadByte(Stream) 
 		End If 
 		 
 	Next 
 	 
 	Return Trim(Name) 
 	 
 End Function 
 
 
 Function SetStringToBitStream(Stream,Name$) 
 	Local NAME_LENGTH=Len(Name)  
 	 
 	WriteByte Stream,NAME_LENGTH 	 
 	Local Bit 
 	Local Char$ 
 	Local Byte 
 	Local Character 
	Local CharBit 
 	Local CharBitValue 
 	Local ByteBit 
 	Local ByteBitValue 
 	Local CharCount 
	 
	For CharCount=1 To NAME_LENGTH 
	Char=Mid(Name,CharCount,1) 
	Character=Instr(CHARSET,Char)-1 
	 
	If (Character<0) 
		RuntimeError "Invalid Character "+Char 
	End If 
	 
	For CharBit=0 To BITS_PER_CHAR-1 
			CharBitValue=(2^CharBit) Mod POSSIBLECHARS 
			ByteBitValue=(2^ByteBit) Mod (2^BYTESIZE) 
			 
			If (Character And CharBitValue) 
				Byte=Byte+ByteBitValue 
			End If 
			 
			Bit=Bit+1 
			ByteBit=Bit Mod BYTESIZE 
			 
			If (Not(ByteBit)) 
				WriteByte Stream,Byte 
				Byte=0 
			End If 
		Next 
		 
	Next 
	 
End Function



virtlands(Posted 2016) [#8]



RemiD(Posted 2016) [#9]
I don't really understand the reason of the clash either, but apparently some steam had to go out and you were the one to take it ;)


_PJ_(Posted 2016) [#10]
""[ I'll be back later to take a closer look at your project. ] ""

This irks me.

My project is nobody else's business. I'm not even sure HOW virtlands intends to gain access to look at such work. It's patently rude to assume he can simply stroll along and scrutinise someone else's work and that's not even a consideration of WHY.
BHy what authority does he seek to appoint himself the one to "look closely" at my projects?

He's acting like some nosey schoolteacher hovering over a students' desk - and for what reason?

_

Again, I have no problem with criticism and constructive comments of code I post here, but that line quoted is neither.


virtlands(Posted 2016) [#11]
... ....


virtlands(Posted 2016) [#12]
... Visual output ?


_PJ_(Posted 2016) [#13]

...
... Please put some visual output into your program, so that we have some clue what it is doing:



There's an example included which provides visual output. That being the file written to the relative location.

  Local FileOut=WriteFile("TestFile.dat") 
 SetStringToBitStream(FileOut,"THIS IS A TEST! OK, YOU'RE SURE?") 
 CloseFile FileOut 
 Local FileIn=ReadFile("TestFile.dat") 
 Local Name$=GetStringFromBitStream(FileIn) 
 CloseFile FileIn 
 WaitKey() 
 End



It's not at all complex. It's actually very very basic.

Consider a bank of code (or a string) as a sequence of bytes:

ABYTE|BBYTE|CBYTE|DBYTE|EBYTE... etc.

These bytes can be themselves considered as built up from bits.

ABCDEFGH|ABCDEFHG|ABCDEFGH|ABCDEFGH|ABCDEFGH

However, if the RELEVANT INFORMATION of each byte may be represented in terms of just 5 bits, then the INFORMATION CONTENT is equivalent with:

ABCDE000|ABCDE000|ABCDE000|ABCDE000|ABCDE000...

the 000 representing "junk" or "padding" bits which are really just wasted space.

Therefore, it is possible to compress the space by removing the junk bits and pulling the bits along from the next bytes in sequence to fill the gaps.


ABCDEABC|DEABCDEA|BCDEABCD|E...

So what did take up 5 bytes now takes up just over 3.

Naturally, there are some pitfalls and an amount of care must be taken to customise the routines SPECIFICALLY FOR THE CASE IN WHICH THEY ARE TO BE USED - once again, this is NOT A GENERAL SOLUTION.

The only remotely complex aspect is how to identify the individual 'nibblets' of information and their position relative to the actual bytes. (i.e. in the example here, in the SECOND byte, from the 3rd BIT of that byte: "ABCDE") --- or that there may be a mismatch at the end (i.e. single bit E with a requirement for at least 7 padding bits.

It can be thought of as temporarily considering a different BIT-DEPTH for the memory, where that BIT DEPTH is not restricted to multiples of 4 or 8.





__


I have included comments and used descriptive variable/method names to identify the key areas of the code.

Were I intending this to be something I offer to the Blitzcommunity as solution and include support thereof, I would post in the ocde archives. As it happens, this thread was due to requesting help with the bug in the code which is now fixed and therefore the thread may be disregarded.




_______________________________________________________________________


At heart, this was all for the purposes of compressing data sizes for load/save to disk or more critically, network communication (with a view to packet sizes obviously necessitating the offset against time taken to process), so there's very little that has any impact or relevance in visual display terms.


_PJ_(Posted 2016) [#14]
Then here.

This shows visually that as the bytes are processed from the big bank, the compressed bank is formed much smaller.

Really do not see what benefit this brings whatseover, but since you asked for it, hope it helps...

;EXAMPLE WITH VISUAL OUTPUT FOR VIRTLANDS - NOT SURE WHAT PURPOSE  IT SERVES SINCE THIS IS NOT SUPPOSED TO BE A VISUAL PROCESS WHATSOEVER

Graphics 1024,768,32,2
Const VO_BIG_X=32
Const VO_BIG_Y=32

Const VO_SMALL_X=32
Const VO_SMALL_Y=64

Const VO_BYTE_GRAPHIC_BLOCK=4

Local n=128

Local TestBank=CreateBank(n)

;Fill testexample
For X=0 To n-1
	PokeByte TestBank,x,Rand(0,7)
Next

Local Shrink=ShrinkBank(TestBank)

WaitKey()
End

Function ShrinkBank(BigBank)
	Local Max=BankSize(BigBank)
	Local ShrinkSize=Max*0.375
	Local Bank=CreateBank(ShrinkSize);Shrink to 3 bits instead of 8 bits
	
	Local BigBytes
	Local BigByteBits
	Local IterBigByte
	Local Word
	Local WordBits
	
	Local SmallByte
	Local SmallShort
	
	Local SmallOffset
	
	For BigBytes =0 To Max-1 Step 8
		
		Word=0
		For IterBigByte=0 To 7
			BigByteBits=(PeekByte(BigBank,BigBytes+IterBigByte) And 7)
			WordBits=BigByteBits Shl (IterBigByte*3)
			Word=Word+WordBits
			
			DrawOutput(BigBytes+IterBigByte,SmallOffset,Max-1,ShrinkSize)
			
		Next
		
		SmallByte=Word And 255
		SmallShort=(Word Shr 8)And 65535
		
		PokeByte Bank,SmallOffset,SmallByte
		PokeShort Bank,SmallOffset+1,SmallShort
		
		SmallOffset=SmallOffset+3
		
		DrawOutput(BigBytes+IterBigByte,SmallOffset,Max-1,ShrinkSize)
	Next
	
	DrawOutput(BigBytes+IterBigByte,SmallOffset,Max-1,ShrinkSize)
	
	Return Bank
End Function

Function DrawOutput(Offset,SmallOffset,BB,SB)
	Cls
	Local X
	
	;Labels
	Color 255,255,255
	Text VO_BIG_X,VO_BIG_Y-12,"Padded data progress: %"+Str(Int(Float(Offset*100)/BB))
	Text VO_SMALL_X,VO_SMALL_Y-12,"Compressed bank size: "+Str(SmallOffset+3)+" bytes"
	
	;BuildGrids
	Color 255,0,0
	
	For X=0 To BB-1
		Rect VO_BIG_X+(X*VO_BYTE_GRAPHIC_BLOCK),VO_BIG_Y,VO_BYTE_GRAPHIC_BLOCK,VO_BYTE_GRAPHIC_BLOCK,False
	Next
;	For X=0 To SB
;		Rect VO_SMALL_X+(X*VO_BYTE_GRAPHIC_BLOCK),VO_SMALL_Y,VO_BYTE_GRAPHIC_BLOCK,VO_BYTE_GRAPHIC_BLOCK,False
;	Next
	
	
	;ShowProgress
	Color 0,255,0
	
	;Normal
	Rect VO_BIG_X,VO_BIG_Y,Offset*VO_BYTE_GRAPHIC_BLOCK,VO_BYTE_GRAPHIC_BLOCK,True
	
	;Compressed
	Rect VO_SMALL_X,VO_SMALL_Y,SmallOffset*VO_BYTE_GRAPHIC_BLOCK,VO_BYTE_GRAPHIC_BLOCK,True
	Flip True
End Function