Ideas for hardcoding text files?

BlitzMax Forums/BlitzMax Beginners Area/Ideas for hardcoding text files?

Grisu

(Posted 2006) [#1]

Hi!

In my app I have 300 text files that need to be loaded into an array. This takes some seconds, but I was wondering if I could find a faster way for this, as the database files are growing in number.

So instead of reading each file, is it possible to "hardcode" the texts. So they no longer have to be read out the incbined files?

I coded a small tool to create a blitz sourcecode out of the text files which I then could "import". This failed as I was not able to handle "Return" and '"' chars inside the files for example.

Any ideas?

Grisu

H&K	(Posted 2006) [#2]

Save them as a memory block. Then point the string to the start of each instance?

This is totaly a guess. So dont take it as advice.

Perturbatio

(Posted 2006) [#3]

This failed as I was not able to handle "Return" and '"' chars inside the files for example.

even with ~n and ~q ?

Grisu

(Posted 2006) [#4]

The problem is that when I write "~n" back to the file, there will be a linebreak. When I later import the file back bmx gives me an error!

When I start adding "Chr(13)" "Chr(34)" by hand it really gets ugly. Couldn't make it work so far.

As the result looks like that:

MyArray[1].name="

Hello World.
These Strings should be in one line.
"

Const MAXFOLGEN=135

Print ""
Print "Starting to convert files..."
Print ""
Load_DDF()


Function Get_Listname:String(s:String)
Local ext:String=""

s=Trim(s)
ext=Left(s,3)

For Local i:Int=1 To 7
	ext=ext+" " 
	'Print Len(ext)
Next  
s=ext+Right(s, Len(s)-4)

Return s
End Function


Function Load_DDF()
Local getid: Int', temp: String 
Local name:String
Local headerlen:Int

Global hFile3:TStream = OpenFile("Import/dfa_import.bmx", False, True)

For Local i:Int=0 To MAXFOLGEN-1

    Print "File "+i+"..."

	Local hFile:TStream  = OpenFile("Data/"+i+".ddf", True, False)
	Local hFile2:TStream = OpenFile("Import/DDF_Data"+i+".bmx", False, True)

    Local s: String = ""
	getid=True
    While Not Eof(hFile)
            s=s+Trim(ReadLine(hFile))+"~n" 

            If getid=True Then
              headerlen=Len(s)-1               

              ' Get name for Listboxentry 
              name=Get_Listname(s)   

        	  s="~n"+s
            ' BLITZBUG WORKAROUND^^^^

' 			  Create_Trailername(0,i)

              getid=False 
            EndIf  
    Wend  

    WriteLine hFile2, "MyArray["+i+"].name="+Chr(34)+name+Chr(34)     

    WriteLine hFile2, "MyArray["+i+"].headerlen="+Chr(34)+headerlen+Chr(34)     

    WriteLine hFile2, "MyArray["+i+"].text="+Chr(34)+s+Chr(34)     

    Local pos:Int=Instr(s,"Erscheinungsdatum:",20 ) 
    Local reldate:String=Trim(Mid(s,pos,19+11))

    WriteLine hFile2, "MyArray["+i+"].reldate="+Chr(34)+reldate+Chr(34)     
 

    CloseFile hFile
    CloseFile hFile2

    WriteLine hFile3, "Include "+Chr(34)+"DDF_Data"+i+".bmx"+Chr(34)    

Next 

CloseFile hFile3
End Function

Byteemoz

(Posted 2006) [#5]

When you want to write "~n" (tilde+n) to a file you have to use "~~n". And that should be the result:
MyArray[1]="~n~nHello World.~nThese Strings should be in one line.~n"

-- Byteemoz

Grisu

(Posted 2006) [#6]

Thanks, getting a bit closer.

Adding the '"' at the right places doesn't work.

When I add "chr(34)" inside the string isn't transformed to '"' inside the file... :/

CS_TBL

(Posted 2006) [#7]

Dunno if it's of any relevancy:

I made a text-parser some days ago which reads a string (so could be made inside your code), that string includes colorcodes and newline-codes, and the parser auto-wordwraps, based on the amount of columns in the viewer. The whole bunch is stored inside a bank (as 2d array), with my own banktext viewer I can scroll upwards/downwards and see my text.

This viewer is a graphical object using the blitzfont.

Or is this not the thing you're looking for?

Grisu

(Posted 2006) [#8]

Thanks for the help.

What I'm trying to do is coding a function that saves! a string like that:

' MyArray[1]="Hello Word"+~n+"This is another text"+~n+~n+"Example 1" '

So I can simply include this file and don't have to read all files at the startup of my app.

Maybe it's just too simple for me... :/

CS_TBL

(Posted 2006) [#9]

Make your own parser.. you can include your own commands then. Will take you a day orso.

Main issue is about your output format.. where/how would you want to display your texts?

Grisu

(Posted 2006) [#10]

I don't need my own commands.

It's a textarea field.
But I also need the data in memory for searching and manipulating / formatting the textarea on the fly.

CS_TBL

(Posted 2006) [#11]

I think I somewhat got the idea.. I'll add it to the codearchives I think.. :P

SuperStrict

DebugLog Chr(10)+Chr(10)+txt2string("C:\Pron\illegal\horses\downloadadres.txt","Local MyText$",0,64)+Chr(10)+Chr(10)

End

Function txt2string$(file$,varname$="Local MyText$",ndx:Int=-1,width:Int=80)

Rem
	txt2string$, by CS_TBL
	
	This function converts a textfile to a string.
	
	command:
	
	parsedtext=txt2string$(file$,varname$,ndx,width)
	
	file$		- the full path of a file (e.g. "C:\My Documents\blahblah.txt")
	varname$	- optional, the name of the variable
	ndx			- optional, if >=0 then the variable is an array, and ndx is its index
	width		- optional, the width of the formatted stringparts
	
	If you want to create a new variable, don't forget to add 'Local ' in the varname!
	(e.g. bla$=txt2string(file$,"Local MyText")
	
EndRem

	If file="" Return ""
	
	Local tmp:TStream=ReadFile(file)
	If Not tmp
		tmp=Null
		Return "<no file found>"
	EndIf
	
	Local bank:TBank=LoadBank(file)	
	
	If BankSize(bank)=0
		bank=Null
		Return "<no data found>"
	EndIf
	
	Local text$
	Local decent$
	
	Local t:Int,v:Int
	
	' make the string content
	
	For t=0 To BankSize(bank)-1
		v=PeekByte(bank,t)		
		Select v
			Case 10
				text:+"~~n"
			Case 13
				text:+"~~r"
			Case 34
				text:+"~~q"
			Default
				text:+Chr(v)
		End Select	
	Next
	
	' now split-up into something decent
	
	decent:+varname ' the filename
	
	If ndx>=0 decent:+"["+ndx+"]" ' optional array-index
	
	decent:+"=.."
	
	For t=0 To Len(text)-1
		If Not (t Mod width) ' newline
			If t>0 decent:+Chr(34)+"+.."
			decent:+Chr(10)+Chr(34)
		EndIf
		decent:+Mid(text,t+1,1)
	Next
	
	decent:+Chr(34) ' and the closing quote!
	
	bank=Null
	Return decent
End Function

Grisu

(Posted 2006) [#12]

I get a "bad escape sequence in string" error message when trying to import the output.

Function Load_DDF()

Global hFile3:TStream = OpenFile("Import/dfa_import.bmx", False, True)

For Local i:Int=0 To MAXFOLGEN-1

    Print "File "+i+"..."

	Local hFile2:TStream = OpenFile("Import/DDF_Data"+i+".bmx", False, True)

    Local s:String=txt2string$("Data/"+i+".ddf","MyArray",i)

    WriteLine hFile2, s 
    CloseFile hFile2

    WriteLine hFile3,  "Include "+Chr(34)+"DDF_Data"+i+".bmx"+Chr(34)
Next 

    CloseFile hFile3

End Function 
End

What for are these ".." you insert?!?

CS_TBL

(Posted 2006) [#13]

Can you post the text you want to convert here? I tested a random textfile (history.txt of that Framework Assistant :P) and it worked like a charm.

.. is to continue a line of code on the next line.

like:

DrawRect 16,..
16,..
80,..
32

Esp. with large arguement-names in functions this is handy, but also for complex formulas in draw commandos..

Grisu

(Posted 2006) [#14]

You've got mail. :)

CS_TBL

(Posted 2006) [#15]

Is there a specific file which doesn't work? I tried 3 ddf's, and they all work like a charm.

Sofar I've seen no issues with my code.. :P

hmmmoment, I just discovered I could actually also support TAB .. *updating*

CS_TBL

(Posted 2006) [#16]

hm.. I think I bumped onto something.. dunno what tho..

CS_TBL

(Posted 2006) [#17]

ah! I found it, but it's a bit weird.. there're arguements why my pov is right, and there're arguements why BRL's implementation would be right.

In short: you can't break an escape!
This:

debuglog "first line ~"+"n next line"

is not valid! (and that's exactly what happened here and there)

So, unfortunately my nicely formatted width will have to break, as escapes on the edge of a stringpart will move to the next line..

Grisu

(Posted 2006) [#18]

Can't you take the 80 char wordbreak out?

I only need this to "feed" the array, don't care if it is easier to read. When I have to change stuff, I'll change the source data and simply run the tool again.

CS_TBL

(Posted 2006) [#19]

Well, I could break it out.. it's done purely for cosmetics anyway.

in any case add this in the select v thing:

			Case 9
				text:+"~~t"

Now you'll have tab support!

After this line:

	If ndx>=0 decent:+"["+ndx+"]" ' optional array-index

change

	decent:+"=.."

with

	decent:+"=~q"

comment out the whole For t=0 To Len(text)-1 loop (you never know) and make a new loop after this block:

	For t=0 To Len(text)-1
		decent:+Mid(text,1+t,1)
	Next

I think that should be it.. :P

(edit) NOOOOOOO CS_TBL, DON'T BE SO SILLY

No new for t loop, just:

decent:+text

^______^

Grisu

(Posted 2006) [#20]

Ahm?

Could you post the whole new code again?

So we are 100% to have the same...

CS_TBL

(Posted 2006) [#21]

I'll keep the WIDTH arguement, I still want these cosmetics in an update, but for now you can use this, I guess..

update: Closefile added and commented txt wiped :P

SuperStrict

DebugLog Chr(10)+Chr(10)+txt2string("C:\Bmaxdev\grisu\Data\17.ddf","Local MyText$",-1,64)+Chr(10)+Chr(10)
End

Function txt2string$(file$,varname$="Local MyText$",ndx:Int=-1,width:Int=80)

Rem
	txt2string$, by CS_TBL
	
	This function converts a textfile to a string.
	
	command:
	
	parsedtext=txt2string$(file$,varname$,ndx,width)
	
	file$		- the full path of a file (e.g. "C:\My Documents\blahblah.txt")
	varname$	- optional, the name of the variable
	ndx			- optional, if >=0 then the variable is an array, and ndx is its index
	width		- optional, the width of the formatted stringparts
	
	If you want to create a new variable, don't forget to add 'Local ' in the varname!
	(e.g. bla$=txt2string(file$,"Local MyText")
	
EndRem

	If file="" Return "<no file given>"
	
	Local tmp:TStream=ReadFile(file)
	If Not tmp
		tmp=Null
		Return "<no file found>"
	EndIf
	CloseFile tmp
	
	Local bank:TBank=LoadBank(file)	
	
	If BankSize(bank)=0
		bank=Null
		Return "<no data found>"
	EndIf
	
	Local text$
	Local decent$
	
	Local t:Int,v:Int
	
	' make the string content
	
	For t=0 To BankSize(bank)-1
		v=PeekByte(bank,t)		
		Select v
			Case 9
				text:+"~~t"
			Case 10
				text:+"~~n"
			Case 13
				text:+"~~r"
			Case 34
				text:+"~~q"
			Default
				text:+Chr(v)
		End Select	
	Next
	
	' now split-up into something decent
	
	decent:+varname ' the filename
	
	If ndx>=0 decent:+"["+ndx+"]" ' optional array-index
	
	decent:+"=~q"
	
	decent:+text
	
	decent:+Chr(34) ' and the closing quote!
	
	bank=Null
	Return decent
End Function

Grisu

(Posted 2006) [#22]

TY again.

Testing and sweatting... :)

Grisu

(Posted 2006) [#23]

Btw: What Name would you like to see in the credits?

CS_TBL

(Posted 2006) [#24]

CS_TBL, and if you mention this blitz website ppl can spot me.. tho I wouldn't care anyway. :P

btw, cosmetics fixed, next post will have the width thing fully operational. I will prolly have it an option again, if width<1 then I'll make a string like the one you're using now, if width>=1 then the string will look nice.

CS_TBL

(Posted 2006) [#25]

tadaa!

SuperStrict

DebugLog Chr(10)+Chr(10)+txt2string("C:\Bmaxdev\grisu\Data\17.ddf","Local MyText$",-1,64)+Chr(10)+Chr(10)

End

Function txt2string$(file$,varname$="Local MyText$",ndx:Int=-1,width:Int=0)

Rem
	txt2string$, by CS_TBL
	
	This function converts a textfile to a string.
	
	command:
	
	parsedtext=txt2string$(file$,varname$,ndx,width)
	
	file$		- the full path of a file (e.g. "C:\My Documents\blahblah.txt")
	varname$	- optional, the name of the variable
	ndx			- optional, if >=0 then the variable is an array, and ndx is its index
	width		- optional, the width of the formatted stringparts, width<1 doesn't format
				- in case of formatting, a line *can* incidentally be one character longer due
				- not being allowing to break escapes
	
	If you want to create a new variable, don't forget to add 'Local ' in the varname!
	(e.g. bla$=txt2string(file$,"Local MyText")
	
EndRem

	If file="" Return "<no file given>"	
	
	Local tmp:TStream=ReadFile(file)
	If Not tmp
		tmp=Null
		Return "<no file found>"
	EndIf
	CloseFile tmp
	
	Local bank:TBank=LoadBank(file)	
	
	If BankSize(bank)=0
		bank=Null
		Return "<no data found>"
	EndIf
	
	Local text$
	Local decent$
	
	Local t:Int,v:Int,w:Int
	
	' make the string content
	
	For t=0 To BankSize(bank)-1
		v=PeekByte(bank,t)		
		Select v
			Case 9
				text:+"~~t"
			Case 10
				text:+"~~n"
			Case 13
				text:+"~~r"
			Case 34
				text:+"~~q"
			Default
				text:+Chr(v)
		End Select	
	Next
	
	' now split-up into something decent
	
	decent:+varname ' the filename
	
	If ndx>=0 decent:+"["+ndx+"]" ' optional array-index
	
	If width<1
		decent:+"=~q"
		decent:+text
	Else
		decent:+"=.."
		For t=0 To Len(text)-1
			If Not (w Mod width) ' newline
				If t>0
					If Asc(Right(decent,1))=126 ' an escape we shouln't break?
						decent:+Mid(text,t+1,1)+Chr(34)+"+.."
					Else ' no? just continue then..
						decent:+Chr(34)+"+.."				
					EndIf
				EndIf
				decent:+Chr(10)+Chr(34)
			EndIf
			decent:+Mid(text,t+1,1)
			w:+1
		Next
	EndIf
		
	decent:+Chr(34) ' and the closing quote!
	
	bank=Null
	Return decent
End Function

Grisu

(Posted 2006) [#26]

This may take a while as the changes seem to break all other functions in my app (around 5.000 lines).

CS_TBL

(Posted 2006) [#27]

this would only be if you use any globals with the names of the variables I use, which would be: tmp, text, decent, t, v, w. (since only w was added, w is suspicious)

Grisu

(Posted 2006) [#28]

Nope, it has nothing to do with that.

It's my way using "Instr" to find the place to highlight certain text passages and key words. It's totally screwed.

Maybe a combination with the textarea bugs in the gui.

*a bit frustrated...