KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

A forum for general discussion of the game: Open to all punters


User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

I recently stumbled upon this new piece of software called Real-ESRGAN. It is relatively new, looks like the last major improvement was a year ago.
What it does is up-scaling small images by using trained neural networks and it manages to provide VERY impressive result for images in anime style when "realesrgan-x4plus-anime" is selected. (And when I say "VERY impressive" I actually mean "VERY impressive" and not exaggerating to get your attention.) This seems to be a HUUUGE improvement if you compare it with other algorithms that have existed to upscale anime so far. I expect that you will be to lazy to try it out on your own, so I took the liberty of trying it out with the 800x600 gallery images from KS and attached some of the results of up-scaling them to 3200x2400 here (see attachments of this post).

(EDIT: The images should also be visible below. Look at them at full size through right clicking if possible.)
shizu_fishing_ah.png
shizu_fishing_ah.png (7.43 MiB) Viewed 16451 times
lilly_bedroom.png
lilly_bedroom.png (5.93 MiB) Viewed 16451 times
hanako_goodend.png
hanako_goodend.png (9.97 MiB) Viewed 16451 times
If you are impressed as well you can get the tool itself here if you want to try it out: https://github.com/xinntao/Real-ESRGAN Both the sourcecode and already usable versions prebuilt for Windows and other OSes are there. These are command-line tools that can be run with following arguments: "-n realesrgan-x4plus-anime -o .\outputdir\ -i .\inputdir\" Oh, and your GPU needs to support Vulkan (which should be the case if it supports DirectX11, which should be the case if it's not from the stone-age) or it might run very slowly or not run at all.

What this means is that KS could be EASILY remade into a FHD or even 4K version without the need to actually redraw any images since this tool makes all the pictures bigger automatically and still makes them look veeery veeery good. I guess you might also want to turn on the OpenGL acceleration in the RenPy engine (which is something you already successfully did with the last Macintosh version), since software based rendering might be a bit to slow for such big pictures.
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

... and a side by side comparison of magnified original versus the result so you can see what I am taking about even if you forgot how the original looks like after all these years:
comparison.png
comparison.png (480.88 KiB) Viewed 16436 times
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

Just generated a version up-scaled with the RealESRGANv2-animevideo-xsx4 option for comparison with realesrgan-x4plus-anime (and also because I like the picture :3). The tool mentions that there is also the option RealESRGANv2-anime-xsx4 but it's broken and I can't find the files that are missing but required to make it work. (Ask the tool's dev if interested.)

realesrgan-x4plus-anime:
shizu_flashback-realesrgan-x4plus-anime.png
shizu_flashback-realesrgan-x4plus-anime.png (5 MiB) Viewed 16364 times
RealESRGANv2-animevideo-xsx4:
shizu_flashback-RealESRGANv2-animevideo-xsx4.png
shizu_flashback-RealESRGANv2-animevideo-xsx4.png (8.78 MiB) Viewed 16364 times
comparison:
comparison2.png
comparison2.png (1.45 MiB) Viewed 16364 times
User avatar
Grayest
Posts: 34
Joined: Tue Sep 29, 2020 1:13 pm
Location: USA
Contact:

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by Grayest »

Holy shit, this is honestly quite incredible! The only issue really is the fact that the aspect ratio obviously isn't up to modern standards. But otherwise, this is gooood
What if the real 'Best girl' was the friends we made along the way?
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

Grayest wrote: Mon Mar 21, 2022 8:51 pm Holy shit, this is honestly quite incredible! The only issue really is the fact that the aspect ratio obviously isn't up to modern standards. But otherwise, this is gooood
Right, the aspect ratio has to stay 4:3. The AI (of 2022) can't magically turn 4:3 into 16:9 by inventing some new stuff that isn't there. I think that if the users know that the original game was made during a time when not everyone had a widescreen yet they wouldn't mind this small inconvenience. The game can still run in windowed mode. Also I wouldn't remove the stuff on the top and on the bottom of the pictures just to FORCE the game to become 16:9. IMHO the contents of the game should stay they way they are except for up-scaling that wold make it look better on "bigger" monitors.
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

Before I forget:

It turns out the the tool is a memory eater. Do not be surprised that it doesn't work if your GPU has significantly less than 4 gigabytes of RAM attached to it. Of course you can try anyway. The speed of the GPU should not matter as far as I know, except of course that it might run slower if your GPU is slow.

If you are using a laptop your CPU and GPU will probably share a common set of RAM chips, so you might need to assign more RAM to the GPU in your BIOS/UEFI settings.
User avatar
Oddball
Posts: 3023
Joined: Wed Feb 29, 2012 2:05 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by Oddball »

What exactly does this enhancement due to the regular backgrounds which were purposely already blurry and distorted?
Not Dead Yet
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

Oddball wrote: Wed Mar 23, 2022 9:31 am What exactly does this enhancement due to the regular backgrounds which were purposely already blurry and distorted?
The easiest way to find out is to just take such a background and upscale it. If you can't do it yourself due to some technical reasons put the picture in here and I'll upscale it for you.

EDIT:
I might have misunderstood your question and maybe you want to hear an accurate technical explanation of HOW the tool operates instead of seeing an example of what it does to a specific picture. If this is the case then I am afraid that an answer to your question might not even exist.
While it is possible to look at the individual neurons of a neural network from the perspective of a computer scientist to understand how they operate on fundamental level and how the programmer fit a neural network into his software and which ideas he used in order to train it, FULLY understanding HOW the finished solution turns a small picture into a big one might be impossible.
A trained neural network doesn't look like something that is easy to understand, it looks like an unfathomable mixed serving of spaghetti. (Of course this analogy is simplified and the actual paths in neural networks look more like trees with A LOT of individually weighted branches and not straight ropes with just two ends on each side. This makes a neural network even more complicated than spaghetti.) The computer that processes such a network doesn't NEED to understand it. It just dumbly runs through all the "innumerable" warped paths noodle by noodle and when it's done it gives a result.
The thing about trained neural networks is that the user usually doesn't even try to understand WHAT EXACTLY it does to the data in order to process it. The usual approach is that you first train a neural network with a data set as large as possible (In this case the data set was made of many anime pictures. It is this automated training process that actually adjusts the weights of the individual branches and thus fills the network with all its overwhelming complexity.) and after that you JUST USE the trained network to process some new arbitrary data in hope that you will get a result that is good enough for you. That's it. There just IS no step that you could call "the user understands the network he created" involved in this approach.
Asking HOW EXACTLY a trained neural network turns some new arbitrary data into a result is like asking the question "How does a human brain operate?". Some parts of your brain were created by evolution and other parts of your brain are trained during your lifetime. A biologist might explain to you how individual nerve cells operate or even give you a very simplified overview of the individual areas of the brain, but the EXACT FULL explanation of a full operating brain is a mystery and might remain a mystery forever.

So we can experiment with individual pictures if you want and hope that the results will answer your questions or otherwise I can not help you. Even the dev of this tool who knows exactly what he has programmed would probably give a similar answer. He would be able to explain how he put the individual pieces of the software together in order to create a working solution of course, but he would never be able to FULLY explain the internal machinery of his neural network AFTER he trained it.
User avatar
OtakuNinja
Posts: 1457
Joined: Sat Jun 16, 2012 10:42 pm
Location: Sweden
Contact:

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by OtakuNinja »

I interpret it more as "why is this needed when the game is made blurry on purpose?"
Image
Emibro, Hanabro, [Lilly Lover], Rin Kin, Feminist, Two-timer
(Passively working on my KS YouTube series. Someday...)
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

OtakuNinja wrote: Thu Mar 24, 2022 7:50 am I interpret it more as "why is this needed when the game is made blurry on purpose?"
Just took a picture that looks like your avatar (actually it was taken out of KS, cropped and down-scaled by 2) and applied A LOT OF blur to it (this is what marked as "original" below even though it's not reeeally original), then checked what happens to it when up-scaling it by 4 with different up-scaling methods. In this exceptional case the linear up-scaling turned out producing a better result than AI, because the AI still tries to sharpen it to make it look as close to normal anime as possible. (In this exceptional case the tool is reeeally pushed close to the boundaries of what it can and can't do. It was never INTENDED to be used in such situations.) However I also looked at the ACTUAL picture-files used by KS and only VERY FEW exceptions have strong blur applied to them. So the OVERWHELMING MAJORITY of them shouldn't be affected by the unintentional sharpening and in the case of the few exceptions exceptions can be made. (There might or might not also be some cases where RenPy applies blur by itself, in which case the problem of unintentional sharpening doesn't matter, since RenPy will blur the picture afterwards one way or another.)
So the answer to your question is: "When a picture was made blurry on purpose this is not needed and a different kind of up-scaling (like "linear", "cubic" or "lanczos") needs to be used instead. When the GAME's ENGINE makes a picture blurry on purpose it also is not needed, but there is also no need to make an exception and actively avoid it."
Note that if there are pictures where a part of the picture is blurry and an other part is not blurry then a separate experiment must be performed. There is still the possibility that unintentional sharpening ONLY occurs because the WHOLE picture is blurry. And if you ask me the case where there are ONLY some blurry clouds in the background still looked pretty well when AI did the up-scaling. The blurry clouds STILL looked smeared out. (See the picture with Hisao's face that I already included into my previous messages. Also see the picture with Misha's head.)
comparison3.2.png
comparison3.2.png (1.48 MiB) Viewed 16160 times
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

The game also contains some backgrounds that are closer to photo-realism than to anime. (Also it seems like someone tried to make some of them look a bit like drawn paintings.) But there are different kinds of up-scaling tools based on neural networks for different purposes. And if THIS ONE doesn't provide satisfying results there are many others that can be tried out instead.
THIS ONE should be suited for sprites and pictures/backgrounds that are not blurry and were unambiguously made in anime style (and maybe for some (if not most) of the ambiguous once as well).
I tested it on many more pictures than just those which I shared here (with many more different sources than just KS) and on those that already looked drawn the effect of looking drawn was very well preserved. Those that didn't already look drawn started looking "a bit drawn". But even THIS tool provides some other options to choose from which reduce this effect in those cases where it is undesirable.
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

Example for an up-scaled background from KS:
school_library_ss-a.png
school_library_ss-a.png (8.02 MiB) Viewed 16143 times
comparison4.png
comparison4.png (876.37 KiB) Viewed 16143 times
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

My gut feeling just told me that my list of examples would be incomplete if I didn't include at least one up-scaled screenshot from the actual game to show off how the end result could look like, so here it is. The textbox was disabled on purpose, since the game engine would render it by itself.
KS-screenshot-a.png
KS-screenshot-a.png (5.62 MiB) Viewed 16135 times
comparison5.png
comparison5.png (2.36 MiB) Viewed 16135 times
User avatar
Grayest
Posts: 34
Joined: Tue Sep 29, 2020 1:13 pm
Location: USA
Contact:

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by Grayest »

DerailedOmegaMale wrote: Thu Mar 24, 2022 3:57 pm Example for an up-scaled background from KS:
school_library_ss-a.png
comparison4.png
Oh wow, very neat. I was wondering how it would handle BGs since they are somewhat blurry in some areas. If using this tool on BGs didn't turn out so well, the best thing to do would've been to upscale the raw photos, and then photoshop them again. (all BGs were made from irl images, in case you didn't know.) Of course, the raw photos aren't available to you or I, you'd have to ask the devs - I know Crud took a good handful of them.
What if the real 'Best girl' was the friends we made along the way?
User avatar
DerailedOmegaMale
Posts: 69
Joined: Tue Nov 05, 2013 3:09 pm

Re: KS could be EASILY turned into a 4K remake [CONTAINS SPOILERS!!!]

Post by DerailedOmegaMale »

Grayest wrote: Mon Mar 28, 2022 4:58 pm Oh wow, very neat. I was wondering how it would handle BGs since they are somewhat blurry in some areas. If using this tool on BGs didn't turn out so well, the best thing to do would've been to upscale the raw photos, and then photoshop them again. (all BGs were made from irl images, in case you didn't know.) Of course, the raw photos aren't available to you or I, you'd have to ask the devs - I know Crud took a good handful of them.
IMHO most of the background pictures look as if they were real photos and then someone added a slight post-processing to them to make them look a bit more cartoon-like (in order to make them better fit the games atmosphere I guess).
I wouldn't worry about those, it still works pretty well. There are only a few where additional blur is applied. And you are right, it would be possible to upscale the original pictures if they were available and blur them out afterwards.
But if the WHOLE picture is blurry then "lanczos" up-scaling works just fine and there is no need to use the original.
If a picture has both blurry and sharp areas then there is also a potential solution that might not need the original. You can upscale such a picture with BOTH the "lanczos" and the AI method in order to get two pictures. Then you can mask out the different areas and fluently merge the pictures back together by means of interpolation.

Note that a mask doesn't necessarily have to be rectangular. It can also have a circular shape in case that a picture has a blurry area right in it's middle.
masking.png
masking.png (49.97 KiB) Viewed 16057 times
Post Reply