Forums

Hi all,

Only 7 days to go until the 2023 coding challenge and if you're struggling to fit your assets into your target ROM size, I might have a little something to help you out :)

So long story short: I was looking for utilities to handle PSGs that I could build onto the Docker toolchain I'm working on, I didn't want to put either the JVM or Mono on the image, and reached out to sverx for some guidance, fully intending to use his psgcomp. We got to chatting, one thing led to another and I ended up attempting to write my own compression algorithm to rival the spicy compression ratios that Calindro somehow managed to achieve with psgtool.

It's been a bit of a labour of love (and when I really should be working on other things!) but I'd like to share it with the community in case it's of use.

Installing

It's written in Go and is by default statically linked with no runtime dependencies, and also cross-compiles very easily so you can almost certainly download directly to run on your favourite environment from the automatically produced release binaries.

Or build from source if you like (under MIT license i.e. permissive).

Usage

A few options for usage, but you can use as a more-or-less drop in replacement for psgcomp if you want to, i.e.

retcon-audio psg <infile> <outfile>

Headline benchmarks

It has the best compression ratio compared with psgcomp and psgtool in 7 out of the 11 test files I benchmarked with, with psgtool coming out top in the other 4.
It comes out worst in 1 of the 11 benchmarks, but not by much.

It compresses quicker than psgcomp in 8 out of 11 test files. I can't really compare it scientifically to psgtool since that also converts from VGM, but from casual observation it's somewhat slower than psgtool although relatively close for most of the files.

For one of the test files (Alex Kidd track 4) it's REALLY slow and I haven't figured out why. I'm getting some interesting results from the time utility - the real time is 12 seconds but the user and sys times are both small, so it might be some inefficiency in the way I'm handling the files or something.

Full benchmark results in this google doc.

Other notes

It doesn't convert from VGM so it needs to be used with something like vgm2psg. No immediate plans or need to make it do this functionality right now but maybe one day.

It also includes a fairly rudimentary debugging facility which kind of "plays" a PSG and prints what it's doing to the screen. It was useful while developing the compression algorithm and for comparing outputs between the three tools. There are a few options to control the debug output.

It reads both compressed and decompressed PSGs at the input and you can use it to decompress by simply running it with a compressed input and the --no-compress flag to force uncompressed output.

I've got some more left field ideas for even better compression algorithms but that will have to wait for some other day...

Thanks

Special thanks to sverx for his support, encouragement and inspiration.
Thanks also to Calindro for setting such a high bar with psgtool's compression in the first place, and without which I would have saved many hours of my life chasing down a few bytes xD

Hmm, I guess this may be useful for a certain FM project I’m working on recently (not in time for the competitions!) with some modifications.

Link to binary releases isn't working. I guess it's missing the final 's'.

Also, now I've got to release the fixed psgcomp too. I don't like at all that I'm losing the competition so badly ;)

Thanks for creating this very useful tool! Maxim will likely request a tweaked version later :rolleyes:

sverx wrote

Link to binary releases isn't working. I guess it's missing the final 's'.

Oh weird, thanks for spotting, now corrected.

sverx wrote

Also, now I've got to release the fixed psgcomp too. I don't like at all that I'm losing the competition so badly ;)

I've come to the conclusion that everyone loses in the end when it comes to attempting to optimise PSG compression xD

Maxim wrote

Hmm, I guess this may be useful for a certain FM project I’m working on recently (not in time for the competitions!) with some modifications.

That wouldn't be converting directly from .fur would it? I'm sure I caught some chat about that on Discord. If so, I'm extremely interested in that...

No, converting from .vgm...

UPDATE: Calindro just dropped a new version of PSGTool.

Benchmarks updated, see attached screenshots.

Very close now!! Sorry sverx, you just got nudged out further :'(

before.png (117.97 KB)

after.png (117.38 KB)

willbritton wrote

Sorry sverx, you just got nudged out further :'(

LOL! The funny part is that is not even my compressor's fault, it's the converter that needed a fix (you guys fixed that part already!).

Will post the fixed converter ASAP - in the while I hope everybody is using either Calindro's or willbritton's wonderful tools, they're much better than my own!

OK I figured out how to use zlib in crosscompiling for Windows on Linux (I'm terrible with these things!) so I just committed the 'fixed' converter, which means probably my compressor will be again at the other guys' level too (or at least I hope so!)

edit: btw it's funny how one would expect the same VGM file to convert into the same compressed PSG and instead you get differences. It's probably because of the complexity of finding 'the sure best' compressed data...

Author	Message
willbritton Joined: 06 Mar 2022 Posts: 697 Location: London, UK	New PSG tool retcon-audio (somewhat improved compression) Posted: Mon Mar 20, 2023 11:01 pm Last edited by willbritton on Tue Mar 21, 2023 8:36 am; edited 1 time in total
	Hi all, Only 7 days to go until the 2023 coding challenge and if you're struggling to fit your assets into your target ROM size, I might have a little something to help you out :) So long story short: I was looking for utilities to handle PSGs that I could build onto the Docker toolchain I'm working on, I didn't want to put either the JVM or Mono on the image, and reached out to sverx for some guidance, fully intending to use his psgcomp. We got to chatting, one thing led to another and I ended up attempting to write my own compression algorithm to rival the spicy compression ratios that Calindro somehow managed to achieve with psgtool. It's been a bit of a labour of love (and when I really should be working on other things!) but I'd like to share it with the community in case it's of use. Installing It's written in Go and is by default statically linked with no runtime dependencies, and also cross-compiles very easily so you can almost certainly download directly to run on your favourite environment from the automatically produced release binaries. Or build from source if you like (under MIT license i.e. permissive). Usage A few options for usage, but you can use as a more-or-less drop in replacement for psgcomp if you want to, i.e. retcon-audio psg <infile> <outfile> Headline benchmarks It has the best compression ratio compared with psgcomp and psgtool in 7 out of the 11 test files I benchmarked with, with psgtool coming out top in the other 4. It comes out worst in 1 of the 11 benchmarks, but not by much. It compresses quicker than psgcomp in 8 out of 11 test files. I can't really compare it scientifically to psgtool since that also converts from VGM, but from casual observation it's somewhat slower than psgtool although relatively close for most of the files. For one of the test files (Alex Kidd track 4) it's REALLY slow and I haven't figured out why. I'm getting some interesting results from the time utility - the real time is 12 seconds but the user and sys times are both small, so it might be some inefficiency in the way I'm handling the files or something. Full benchmark results in this google doc. Other notes It doesn't convert from VGM so it needs to be used with something like vgm2psg. No immediate plans or need to make it do this functionality right now but maybe one day. It also includes a fairly rudimentary debugging facility which kind of "plays" a PSG and prints what it's doing to the screen. It was useful while developing the compression algorithm and for comparing outputs between the three tools. There are a few options to control the debug output. It reads both compressed and decompressed PSGs at the input and you can use it to decompress by simply running it with a compressed input and the --no-compress flag to force uncompressed output. I've got some more left field ideas for even better compression algorithms but that will have to wait for some other day... Thanks Special thanks to sverx for his support, encouragement and inspiration. Thanks also to Calindro for setting such a high bar with psgtool's compression in the first place, and without which I would have saved many hours of my life chasing down a few bytes xD

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14984 Location: London	Posted: Tue Mar 21, 2023 7:25 am
	Hmm, I guess this may be useful for a certain FM project I’m working on recently (not in time for the competitions!) with some modifications.

sverx Joined: 05 Sep 2013 Posts: 3951 Location: Stockholm, Sweden	Posted: Tue Mar 21, 2023 8:33 am
	Link to binary releases isn't working. I guess it's missing the final 's'. Also, now I've got to release the fixed psgcomp too. I don't like at all that I'm losing the competition so badly ;) Thanks for creating this very useful tool! Maxim will likely request a tweaked version later :rolleyes:

willbritton Joined: 06 Mar 2022 Posts: 697 Location: London, UK	Posted: Tue Mar 21, 2023 8:41 am
	sverx wrote Link to binary releases isn't working. I guess it's missing the final 's'. Oh weird, thanks for spotting, now corrected. sverx wrote Also, now I've got to release the fixed psgcomp too. I don't like at all that I'm losing the competition so badly ;) I've come to the conclusion that everyone loses in the end when it comes to attempting to optimise PSG compression xD Maxim wrote Hmm, I guess this may be useful for a certain FM project I’m working on recently (not in time for the competitions!) with some modifications. That wouldn't be converting directly from .fur would it? I'm sure I caught some chat about that on Discord. If so, I'm extremely interested in that...

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14984 Location: London	Posted: Tue Mar 21, 2023 10:50 am
	No, converting from .vgm...

willbritton Joined: 06 Mar 2022 Posts: 697 Location: London, UK	Posted: Tue Mar 21, 2023 10:33 pm
	UPDATE: Calindro just dropped a new version of PSGTool. Benchmarks updated, see attached screenshots. Very close now!! Sorry sverx, you just got nudged out further :'( before.png (117.97 KB) after.png (117.38 KB)

sverx Joined: 05 Sep 2013 Posts: 3951 Location: Stockholm, Sweden	Posted: Wed Mar 22, 2023 12:58 pm
	willbritton wrote Sorry sverx, you just got nudged out further :'( LOL! The funny part is that is not even my compressor's fault, it's the converter that needed a fix (you guys fixed that part already!). Will post the fixed converter ASAP - in the while I hope everybody is using either Calindro's or willbritton's wonderful tools, they're much better than my own!

sverx Joined: 05 Sep 2013 Posts: 3951 Location: Stockholm, Sweden	Posted: Thu Mar 23, 2023 3:40 pm
	OK I figured out how to use zlib in crosscompiling for Windows on Linux (I'm terrible with these things!) so I just committed the 'fixed' converter, which means probably my compressor will be again at the other guys' level too (or at least I hope so!) edit: btw it's funny how one would expect the same VGM file to convert into the same compressed PSG and instead you get differences. It's probably because of the complexity of finding 'the sure best' compressed data...

Forums

View topic - New PSG tool retcon-audio (somewhat improved compression)