As a small amateur project, I experimented with the AI Gigapixel neural networks to upscale one of my favorite science fiction series - Star Trek: Deep Space Nine (DS9), in the Russian translation of Star Trek: Deep Space 9.

Prospects for Deep Space Nine

Just like Final Fantasy 7, where I do an upscale of backgrounds, textures and videos in the Remako mod , DS9 has no plans to release an HD version. While the popular seasons Original Series and The Next Generation (TNG) were mostly filmed, the DS9 was filmed in the mid-90s with visual effects (space battles, etc.).

An analog film can be re-scanned with a higher resolution, but this will not work with digital video. Remastering the series is becoming much more expensive: this is one of the reasons why it was not released in HD.

DS9 remastered machine learning

I thought that it would be useful to use neural networks here. Using tools like AI Gigapixel, DS9 low-resolution frames can be scaled to high definition: 1080p or 4K. They will never compare with a good remastering, but this is a step in the right direction.

Therefore, I tried on a couple of frames what happens. The results are great. AI Gigapixel uses neural networks trained in real photos. They are well suited for the upscale renders of Final Fantasy video games, but they also scaled DS9 frames and special effects with amazing quality.

Here are some examples: Original resolution 480p (click to enlarge) Improved resolution 1080p (click to enlarge) Original resolution 480p (click to enlarge)











Enhanced 1080p resolution (click to enlarge)

These individual frames looked promising. In the first set of images, the station staff in spacesuits represents only a few spots. After the upscale, much sharper figures appeared.

The close-up of the arm also improved. The wrinkles and creases on the skin look much more detailed, and the baseball shows better shine and intricate firmware.

Video

However, the real check is how the neural networks will cope with the sequence of frames, that is, video. Will artifacts and other problems appear? In the end, Gigapixel AI is designed to upscale individual images and does not take into account the relationship between individual frames of a video.

For verification, I decided to process part of the episode "Victim of the Angels" of the 6th season. This is a great series about the Dominion War, where there are epic space battles and more personal moments face to face.

I will tell you more about this process in future articles, but in short: extracting and scaling frames, and then generating a new video from them, took about two days. This is the processing of the first five minutes of the episode (resume, opening scene and introduction). Quite a normal time for mid-range and high-end PCs with open source software.

The result led me into awe. He looked better than I hoped. No problems and artifacts. Since (moving) images are worth more than a thousand words, here are two videos for comparison.

The first demonstrates the “before and after” situation on still frames.


The second compares two videos. Notice how sharper the improved version looks.


I highly recommend watching these videos through the YouTube app on your TV, if possible. This gives a better feeling of how the enhanced DS9 will look directly on TV.

Comparisons are good, but what does a real video look like? Here are the first five minutes of the episode in full with a resolution of 1080p :

What about 4K?

Honestly, I don’t know. Although I can zoom in to 4K, I don’t have a TV or monitor with that resolution to check for improvements. However, I recorded this video. It is interesting to hear the opinion of people with 4K equipment whether it looks better compared to the 1080p version.

Rendering almost melted my computer, since it is much more intensive than 1080p, so we will limit ourselves to this 4K example.

What's next?

Since I do not own the rights to DS9, I can’t do what I want with the series. I would release full episodes, but it is just legally impossible. These videos are more of a proof of concept for CBS copyright holders to take a look at machine learning and neural networks for a possible remastering of DS9, which will bring our favorite movie a bit closer to the HD era.

Imagine what a professional team can do with powerful equipment, specially trained neural networks (perhaps by training it in the TNG season - on a set of data from original frames and frames after remastering) and access to SD sources instead of DVDRip, like mine.

Well, for my part, I can tell in more detail about the upscale process, how I conducted it, which will be the topic of a future article.

Let me know what you think about it.