Saving Type Fields
Blitz3D Forums/Blitz3D Programming/Saving Type Fields
| ||
I've got a rather large database comprised of nested types that I want to save to a file. The structure looks like this :
type Universe
field Galaxy_info[36]
end type
type Galaxy_info
field system.system_info[500]
end type
type System_info
field Planet.Planet_info[15]
end type
type planet_Info
field (about 40 various fields)
end type
Using For/Next loops I can manage to save it all, but it takes a rather long time. I've never waited on the entire save, only going so far as about 2 min and getting to galaxy 15 before halting the code. I'm writing a program to manage game information and taking that long to save the file, especially if only some of it has been updated, just isn't going to work. Is there some way that I can save it faster AND be able to access specific portions of it without having to resave the entire thing if some of the data changes? Here is the current save code I'm using
Function SaveUniverseInfo()
;savefile$ = "Universe.dat"
savefile = WriteFile(UniverseSave)
WriteString savefile, SaveFileVersion
For gal = 1 To TOTALGALAXIES ;36
For sys = 1 To TOTALSYSTEMS ;499
For pla = 1 To TOTALPLANETS ;15
p.planet_info = PLANETget(gal,sys,pla)
WriteString savefile, p\PlanetName
WriteString savefile, p\playername
WriteInt savefile, p\imageNum
WriteInt savefile, p\galaxy
WriteInt savefile, p\system
WriteInt savefile, p\planet
WriteInt savefile, p\slots
WriteInt savefile, p\slotsused
WriteInt savefile, p\ore
WriteInt savefile, p\cry
WriteInt savefile, p\hyd
WriteInt savefile, p\oremine
WriteInt savefile, p\crymine
WriteInt savefile, p\hydmine
WriteInt savefile, p\yard
WriteInt savefile, p\cap
WriteInt savefile, p\lab
WriteInt savefile, p\silo
WriteInt savefile, p\factory
WriteInt savefile, p\orehouse
WriteInt savefile, p\cryhouse
WriteInt savefile, p\hydhouse
WriteInt savefile, p\foundry
WriteInt savefile, p\laser
WriteInt savefile, p\armor
WriteInt savefile, p\weap
WriteInt savefile, p\shield
WriteInt savefile, p\part
WriteInt savefile, p\jet
WriteInt savefile, p\ai
WriteInt savefile, p\energy
WriteInt savefile, p\spy
WriteInt savefile, p\pulse
WriteInt savefile, p\plasma
WriteInt savefile, p\ftl
WriteInt savefile, p\exped
WriteInt savefile, p\warp
WriteInt savefile, p\arc
WriteInt savefile, p\atla
WriteInt savefile, p\sats
WriteInt savefile, p\herc
WriteInt savefile, p\arte
WriteInt savefile, p\apol
WriteInt savefile, p\char
WriteInt savefile, p\pose
WriteInt savefile, p\athe
WriteInt savefile, p\hade
WriteInt savefile, p\prom
WriteInt savefile, p\zeus
WriteInt savefile, p\ares
WriteInt savefile, p\dion
WriteInt savefile, p\herm
WriteInt savefile, p\gaia
Next
Next
Next
;CloseFile(savefile) <--- not sure if I need this?
End Function
It's been a few years since I coded in Blitz, but I keep thinking there was a way to save an entire type to a file - and I was thinking that you could also write to a specific spot so you could just alter specific changes rather than rewriting the entire save file. |
| ||
| Unfortunately the only way to dump a type in one go is with Str(). That probably won't help you much here. A file stream has a "pointer", that's incremented each time you read from or write to the stream. You can set it to a known location with SeekFile. If all your data structures are of a predictable size, this might help you update data - but you're saving strings as well, so you'll have to take their varying sizes into account. The fastest way to get large amounts of data in and out of a file is with WriteBytes and ReadBytes (not to be confused with Read/WriteByte singular!). These copy data into or out of a bank. The fact you're only using one read/write operation makes things a whole lot faster - you can then get at the data in the bank with Peek and Poke as normal; it'll all be in the same locations in the bank as it was in the file. And yes, you do need to CloseFile(), unless you want to chew through your memory in a hurry and corrupt your painstakingly-saved data. |
| ||
| If i were you i would write a set of functions to serialise and deserialise the contents into a tagged structure. That way you can change the format at any time without corrupting the files. (that is the big disadvantage of binary files) Also, dont worry about it being human readable, it is EASY to apply switchable encryption to both the serialiser and deserialiser when it comes to release time. ie example data file:
#---------------------------------
[Galaxy]
GalaxyID=1
Name = Milky Way
Stars = BloodyLoads
(all data for this object)
#---------------------------------
[System]
SystemID=1001
Galaxy_ID=1
Name = blahblah
Size = blahblah
Alleigence = baddies
(all data for this object)
[Planet]
PlanetID=45645644
System_ID=1001
Galaxy_ID=1
Name=whatever
gonvernment=anarchy
Poplation =whatever
(all data for this object)
[/Planet]
[Planet]
PlanetID=12345643
System_ID=1001
Galaxy_ID=1
Name=whatever2
gonvernment=anarchy
Poplation =whatever
(all data for this object)
[/Planet]
[/System]
#---------------------------------
[System]
SystemID=1002
Galaxy_ID=1
Name = another system
Size = blahblah
Alleigence = baddies
(all data for this object)
[Planet]
PlanetID=62345643
System_ID=1002
Galaxy_ID=1
Name=whatever
gonvernment=anarchy
Poplation =whatever
(all data for this object)
[/Planet]
[Planet]
PlanetID=65345643
System_ID=1002
Galaxy_ID=1
Name=whatever2
gonvernment=anarchy
Poplation =whatever
(all data for this object)
[/Planet]
[/System]
[/Galaxy]
#---------------------------------
[Galaxy]
GalaxyID=2
Name = Andromeda
.
.
.
.
blah.
blah.
blah.
etc.... OR, even better use XML (although it is a little bit more involved than writing your own, however the advantages are exponential. PS: You will find this little gem very useful (you can use it to split lines, symbols and values really easily when parsing:
;-----------------------------------------
;USV - universally separated values - This is a handy little tool i wrote, great for scipts for returning
;values from a String separated by any character you like. Defaults to a comma but can be anything
;eg "-" "_" "=" "|" etc...
;use: value=USV$("10,20,30" ,2 [separator$=","] ) would return 20
;-------------------------------------------------------
Function Misc_USV$(in$,which%=1,sep$=",")
Local n% = 1
Local offset% = 0
Local nextoffset% = 1
Local ValueRet$ =""
While offset<Len(in$)
nextoffset = Instr(in$,sep$,offset+1)
If nextoffset = 0
nextoffset = Len(in$)+1
which = n
End If
valueret$ = Mid$(in$,offset+1,nextoffset-offset-1)
If which = n
Return valueret
End If
offset = nextoffset
n=n+1
Wend
Return n-1
End Function
If you are interested, take a look at my EPR loader for Ted, it does something similar. (check out the "instances" and "custom" properties sections) |
| ||
| You need some groupings and flags. like (is civilized), (has constructs), (uses energy) or some such to allow you to skip saving batches of data. Group whole systems into a hierarchy as well, so a cluster of stars has the highest cmplexity max of the highest planet. Then you won't be reading & writing so many 0's. |
| ||
| You can add as many flags as you want to the structure above, if data is not needed(ie, isPopulated=false), then you dont need to include data lines like population, government or exports/imports etc. |
| ||
| Thanks for all that info guys. I'll study it and see if I can figure it out enough to implement and try. Question for D4NM4N, are those codes for Blitz2d/3d? I don't recognize the usage of code like [Planet] . . [/Planet] I'm afraid that I don't follow Yasha well. Not familiar with Banks at all and not too sure with the usage of peek and poke (tho I did use those commands way back (like 30 years ago!) when I programmed in Atari Basic) Still, lots to consider. Thanks again! |
| ||
| They are -your- codes. They can be whatever you want them to be. For example, this is from a reader i made a while ago: Here is an example of reading one of the chunk of data in above function: Notice this one above is only 1 level deep. However, there is nothing to stop you recursing as many "sub chunks" as you like eg:
Function Readchunk_SKY(file)
chname$="SKY"
sky.SkyType = New SkyType
Repeat
linein$=ReadLine(file)
key$=Misc_USV(linein$,1,"=")
value$=Misc_USV(linein$,2,"=")
key$=Replace(key$," ","")
DebugLog chname$+": "+Key+" = "+value
Select key$
Case "TED_Sky_Up" ;read value
Case "TED_Sky_Dp" ;read value
Case "TED_Sky_Le" ;read value
Case "TED_Sky_Ri" ;read value
etc.....
Case "[BIRD]"
bird.BirdType = New BirdType
Repeat
linein$=ReadLine(file)
key$=Misc_USV(linein$,1,"=")
value$=Misc_USV(linein$,2,"=")
key$=Replace(key$," ","")
DebugLog chname$+": "+Key+" = "+value
Select key$
Case "BirdType" ;read value
Case "BirdColor" ;read value
etc...
EndSelect
if eof(file) debuglog("Error: Missing terminator on bird chunk!"
Until Instr(linein$,"[/SKY]")
EndSelect
if eof(file) debuglog("Error: Missing terminator on sky chunk!"
Until Instr(linein$,"[/SKY]")
end function
A better way than what i have done is use While not EOF(file) rather than repeat i guess, but it worked for me because i did not want the app to "survive" anyway if there was an error in the read. To create the file you just do the above in reverse with writelines and for-loops. Or just use a text editor! (Simple! :) |
| ||
You also might want to try something like this:
Function SaveUniverseInfo()
;savedir$ = "Universe.dat"
CreateDir( UniverseSave )
;save the file version
savefile=writefile(UniverseSave+ "\filever.txt")
WriteString savefile, SaveFileVersion
closefile(savefile)
;save the information
For gal = 1 To TOTALGALAXIES ;36
For sys = 1 To TOTALSYSTEMS ;499
For pla = 1 To TOTALPLANETS ;15
p.planet_info = PLANETget(gal,sys,pla)
SavePlanet(p, UniverseSave+ "\" +gal+ "-" +sys+ "-" +pla+ ".txt")
Next
Next
Next
End Function
Function SavePlanet(p.planet_info, filename$)
file=writefile(filename$)
;SAVE THIS PLANET'S INFO
closefile(file)
End Function
This saves all of the planets in a folder, with each planet having it's individual file. So a planet in galaxy 6, system 5, and planet 1 would be saved in the file "6-5-1.txt" The advantage to this approach is that you can edit a planet easily without loading the entire universe. The downside is that you have a bunch of little files to keep track of. |
| ||
| Thanks for the replies everyone! Sorry I couldn't respond sooner. Holiday travels. I think the biggest problem I'm having is the volume of information I need to save. 36 x 499 x 15 = 269,460 separate planet records. And that's just for now. If the game increases in size then more galaxies will be added. The program I'm writing is just to keep track of game information - who you probed, what's on planets you know about, etc - and that information changes daily. Rewriting and loading the full data file is just too much. Too much time and too much memory for that. Most of that information is just going to be zero's anyway because the scope of the environment is just too big for any 1 player to scan it all. Perhaps xtremegamer's example is the way to go. For the most part the information is going to be gained over time and build up and being able to pull up individual file information will make it fast. Over time that could create a lot of text files, but they will all be pretty small in size, again, making reading and writing fast. I think for the moment I'm going to go that route so I can get going on the rest of the data sifting and stuff I want to get done. The hold up was saving information and I see now that trying to save and load the entire universe is foolish - especially if a player is only going to ever access less than 1% of it. Thank you so much for all your help and direction! And happy holidays! |
| ||
| If you want to update only portions of a file, it is easiest to ensure every piece of information has a fixed length. Integers and Floats are allways 4 bytes, but Strings can vary their length. Once you know which value you want to update, and you know it's location in the file, you can use the OpenFile to open an existing file, SeekFile to look up the portion you want to update, and then WriteInt/WriteFloat/WriteByte etc to write the data. OpenFile does not create a new file. You need WriteFile to do that. But that shouldn't be a problem, since you can use FileType to determine if the file exists, and then either use WriteFile or OpenFile. First step however, would be unifying the size of each element. For instance, allways use 64 character-strings. |
| ||
| I'd considered doing this with a png file. That way I could just create an image and save it and it would be the default universe, then each pixel I could save 3 values with, 1 stored in red, one in blue, and one in green. I'd know how many pixels represented each value, setting aside a max amount for strings, and I could just read and write to the pixels. with the color values ranging from 0 to 255 I could easily store values or asc values in there. Galaxy 1, system1, planet 1 would start at pixel 1 and use up enough pixels for each of the type values. then Planet 2 would start where that ended - each planet taking up a pre-specified chunk of the image's pixels. If I had a word "dot" then the first pixel colors would reflect the asci values of that word. Set red to asc("d"), blue to asc("o"), and green to asc("t"). If I allowed for a maximum string length of 21 letters then I would reserve 7 pixels. With numbers it would be about the same. Values that would be larger than 254 could be broken down. 1 pixel could hold a value of 999,999 with each color holding the num 99 - 99(red)99(blue)99(green). I'm sure there are better ways to do it, but in the sense of getting on with the programming it would be a method that I would inherently understand rather than spending the time to learn how to poke and peek or learn banks. |
| ||
| I still think that unless there is going to be a LOT of data (im talking LOADS, more than you mention above) then text based files are far more dynamic, safer, easy to debug and version proof. Once encrypted they offer as good protection as straight binary writes. (The only real advantage to mixed type 'binary' files (created with writebyte, writeint, writestring etc) is io speed and size, but i doubt that applies here). |
| ||
| I've run into a real weird problem now. The program runs fine, I'm saving the info in individual files, able to pull them up, and use them. Working on displaying the info and then suddenly I get a memory access violation error. It happens when I add in the next text line. The syntax is good and the error occurs on an image not existing that is right there in memory. If I take out that text line then it all works fine. My best guess is that somewhere I didn't close a save file or something, although on my initial examination I don't see that. Going to have to go through with a fine tooth comb I guess. Sigh. I hate wasting time on things like this. |
| ||
| Code? It usually means you are writing or reading a wrong data type or are overrunning your items dataarea or hitting EOF. This is what I meant about bin files being non-dynamic. The data must be read in the exact same typeorder it was written. (a big problem if you may be adding or removing data items and changing your mind throughout development. It also means you need to write data for every field even if it is not needed (eg has value zero, empty or false)) Sigh. I hate wasting time on things like this. Then use a tagged structure like i said :DIf you give me the type specs i can help you get started on a writer/reader. It will save you hours of headscratching and you can even edit/tweak your universes in wordpad while the files are unencrypted. |
   |