Nethack Round-robin #2

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

Quoting Douglas Henke <henke@kharendaen.dyndns.org>:
>rlb@hoekstra-uitgeverij.nl (Richard Bos) writes:
>>It would make the save files several times larger, and they're large
>>enough as it is for some of the smaller platforms NetHack runs on.
>That may be a legitimate concern, though I question whether anyone plays
>much these days on any platform more cramped than, say, Wince on PocketPC.

On my Psion, a 52-level savefile - created by going into wizard mode and
^Ving to each level, so lacking branches and the many objects created by
actually playing - is 427Kb, with the machine presently having 7.2Mb free.
A large increase in savefile size would definitely present a problem.
--
David Damerell <damerell@chiark.greenend.org.uk> Kill the tomato!
Today is Gloucesterday, April.

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

David Damerell <damerell@chiark.greenend.org.uk> writes:
> On my Psion, a 52-level savefile - created by going into wizard mode and
> ^Ving to each level, so lacking branches and the many objects created by
> actually playing - is 427Kb, with the machine presently having 7.2Mb free.
> A large increase in savefile size would definitely present a problem.

Does your Psion build keep the save/level/bones files compressed?
(I.e., does it define COMPRESS in config.h? If so, what is the value?)

If no, would doing so cause an unacceptable (=="perceptible") delay
in level changes?

(If so, going from gzipped binary saves to (say) gzipped XML would
probably increase size-as-stored by only a few percent. That's assuming
one were moderately clever about schema design...)

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

Douglas Henke <henke@kharendaen.dyndns.org> wrote:

> rlb@hoekstra-uitgeverij.nl (Richard Bos) writes:
> > Douglas Henke <henke@kharendaen.dyndns.org> wrote:
> > > [ cross-platform portable saves, levels and bones ]
> > > Which is why it will be necessary to invent XML, XDR or plain old flat
> > > ASCII... only someone already has.
> >
> > Nah... they're human-readable, and therefore too easily modifiable with
> > just a text editor.
>
> This is why, in the year 2005, we have things like digital signatures
> and operating systems with proper file permissions. If you want to stop
> people messing with the save files, then stop them doing that

On MS Windows? On a hand-held?

> > It would make the save files several times larger, and they're large
> > enough as it is for some of the smaller platforms NetHack runs on.
>
> That may be a legitimate concern, though I question whether anyone plays
> much these days on any platform more cramped than, say, Wince on PocketPC.

I wouldn't play anything on the all too aptly named Wince, but there are
people playing NetHack on Psions.


> > A fixed binary file format, in which the values of the game data have
> > been marshalled into an explicit number of bytes in an explicit order,
> > is just as simple as a textual format - in some ways easier - and much
> > safer and smaller.
>
> 1) The binary format used by the game today forces neither a specific
> number of bytes nor a specific order. Word length, byte order
> within multibyte integers, and padding of in-memory structures
> (dumped directly to file) are at the whim of the compiler and
> options chosen for a particular build.

You write this as if you think I don't know it.

> 2) Development and debugging are _much_ easier if you have human-readable
> files, and can leverage libraries other people have already written
> and tested (e.g. for parsing XML).

There are libraries like that for fixed-format binary files as well as
for XML. They tend to be simpler, too.

> 3) If the size of files is a big deal, you know where bzip2 is.

Does bzip2 compress a stream of bytes on the fly, while writing the
file? Because it's no good first writing five megs of save file and
_then_ compress it if you haven't got that much space in the first
place.

Richard

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

Quoting Douglas Henke <henke@kharendaen.dyndns.org>:
>David Damerell <damerell@chiark.greenend.org.uk> writes:
>>On my Psion, a 52-level savefile - created by going into wizard mode and
>>^Ving to each level, so lacking branches and the many objects created by
>>actually playing - is 427Kb, with the machine presently having 7.2Mb free.
>>A large increase in savefile size would definitely present a problem.
>Does your Psion build keep the save/level/bones files compressed?
>(I.e., does it define COMPRESS in config.h? If so, what is the value?)

I hope Duncan Booth will answer this, because I cannot...
--
David Damerell <damerell@chiark.greenend.org.uk> flcl?
Today is Leicesterday, April.

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

David Damerell <damerell@chiark.greenend.org.uk> writes:
> >Does your Psion build keep the save/level/bones files compressed?
> >(I.e., does it define COMPRESS in config.h? If so, what is the value?)
>
> I hope Duncan Booth will answer this, because I cannot...

If you're violently curious, you might be able to tell by looking at
the files themselves. Alternatively, if you'd like, email me one of
the level files (or post a link to ditto) and I'll have a look.

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

David Damerell wrote:

> Quoting Douglas Henke <henke@kharendaen.dyndns.org>:
>>David Damerell <damerell@chiark.greenend.org.uk> writes:
>>>On my Psion, a 52-level savefile - created by going into wizard mode
>>>and ^Ving to each level, so lacking branches and the many objects
>>>created by actually playing - is 427Kb, with the machine presently
>>>having 7.2Mb free. A large increase in savefile size would definitely
>>>present a problem.
>>Does your Psion build keep the save/level/bones files compressed?
>>(I.e., does it define COMPRESS in config.h? If so, what is the value?)
>
> I hope Duncan Booth will answer this, because I cannot...

No, it doesn't define COMPRESS. The COMPRESS option requires the presence
of an external compression program and I couldn't be bothered to port one.
There was a version of gzip for the Psion at one point, but that would have
meant adding a dependancy which might have discouraged some people from
installing nethack.

I don't know if the Psion gzip is still available, nor how much memory it
took.

Douglas Henke wrote:
>
> If no, would doing so cause an unacceptable (=="perceptible") delay
> in level changes?

Possibly. Compressing the save files involves writing the uncompressed file
to disc and then spawning a separate program to do the compression. The
limiting factor on the Psion, at least if you use an external flash drive,
is the time taken to write the file to the flash card. OTOH, If the file
was written to RAM and compressed to the flash card it might actually be
faster.

One of the slightly odd things about the Psion 5 was that it was known for
being a fast machine (compared to the WinCE pdas of that time) even though
it had a much slower processor than the equivalent WinCE machines. This was
largely because of the way the software and the OS were written. An
unfortunate side effect was that if you actually wrote a CPU intensive
program you suddenly found just how slow the processor could be. I expect
that compressing the save files would be noticeable on a Psion 5 but much
less so on a Netbook.

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

Duncan Booth <duncan.booth@invalid.invalid> writes:
> David Damerell wrote:
> >>Does your Psion build keep the save/level/bones files compressed?
> >>(I.e., does it define COMPRESS in config.h? If so, what is the value?)
> >
> > I hope Duncan Booth will answer this, because I cannot...
>
> No, it doesn't define COMPRESS. The COMPRESS option requires the presence
> of an external compression program and I couldn't be bothered to port one.

From src/include/config.h:
* If COMPRESS is defined, it should contain the full path name of your
* 'compress' program. Defining INTERNAL_COMP causes NetHack to do
* simpler byte-stream compression internally.

The INTERNAL_COMP stuff seems to be dead-simple (look for RLECOMP and
ZEROCOMP in the sources) and would probably work without a major speed
penalty on palmtops. (Or did you try this already and find it unsuitable?)

On the gripping hand, this isn't the sort of compression that would
bring (e.g.) XML level files within a few percent of packed binary.

> Possibly. Compressing the save files involves writing the uncompressed file
> to disc and then spawning a separate program to do the compression. The
> limiting factor on the Psion, at least if you use an external flash drive,
> is the time taken to write the file to the flash card. OTOH, If the file
> was written to RAM and compressed to the flash card it might actually be
> faster.

Things like zlib can compress streams on the fly without using temporary
files. They like big (in the case of bzip2, *very* big, like 900K) buffers,
though, which might be a deal-killer. It also wouldn't be a drop-in
replacement for the UNIX-style command-line "compress the temp file"
stuff in the 3.4.3 sources.

In addition to the "replace slow IO with less-slow CPU crunch", there's
something to be said for not using up the finite lifetime (in write-cycles
per cell) of your flash on something avoidable...

> One of the slightly odd things about the Psion 5 was that it was known for
> being a fast machine (compared to the WinCE pdas of that time) even though
> it had a much slower processor than the equivalent WinCE machines.

s/Psion 5/PalmOS/. BTDT, GTTS. That's the beauty of Wince; you can compare
pretty much anything you like to it, and that other thing is going to look
like a winner.

 

[Guest]

Archived from groups: rec.games.roguelike.nethack (More info?)

Douglas Henke wrote:

> The INTERNAL_COMP stuff seems to be dead-simple (look for RLECOMP and
> ZEROCOMP in the sources) and would probably work without a major speed
> penalty on palmtops. (Or did you try this already and find it
> unsuitable?)

Sorry, I thought the question was about the COMPRESS option. Yes, the Psion
Nethack already uses INTERNAL_COMP, but as you say:
>
> On the gripping hand, this isn't the sort of compression that would
> bring (e.g.) XML level files within a few percent of packed binary.
>

> Things like zlib can compress streams on the fly without using
> temporary files. They like big (in the case of bzip2, *very* big, like
> 900K) buffers, though, which might be a deal-killer. It also wouldn't
> be a drop-in replacement for the UNIX-style command-line "compress the
> temp file" stuff in the 3.4.3 sources.

I wasn't saying that it wouldn't be possible to use zlib to compress the
save files, just that the current COMPRESS support requires writing to a
temporary file which has implications on a platform like the Psion. If the
basic nethack code had used zlib directly I might well have used it.

Even spawning an external non-gui application with command line options is
non-trivial on the Psion (not hard, just badly documented).

>
> In addition to the "replace slow IO with less-slow CPU crunch",
> there's something to be said for not using up the finite lifetime (in
> write-cycles per cell) of your flash on something avoidable...
>
Good point.