Dual GPU Passthrough on Proxmox

This post explains how I managed to pass through some old NVIDIA GTX1060 cards to two Rocky Linux VMs on Proxmox 9. One of those virtual machines is even able to make use of a local monitor and keyboard/mouse while the other VM will be for remote desktop access only. This was pretty straightforward in hindsight except that tutorials on the web were often a few years old and did different things, so I was unsure at first which pieces of advice were still valid.

Skip down to “How To” if you’re in a hurry.

In VFX studios people who usually are not working on beefy workstations might still have to play back 4K video or review full-resolution files (OpenEXR in our case). While mini PCs (HP EliteDesk, Zotac, Forum Mini and what not) with often just 16GB of RAM and onboard graphics are more than enough for some spreadsheet and database work, the playing back of video once or twice a day would really benefit from more power.

Setup

We had an old HP Z820 workstation around and old GPUs that were no longer up to the task for actual VFX work in Nuke. The workstation, however, has dual 16 core CPUs, 128 GB or RAM and server-grade hardware inside (I don’t get any money from HP for saying this but I like their workstations and we’ve been using refurbished models from Z800 to Z8G4). There’s enough room to fit 2 consumer-grade graphics cards from MSI in there and the PSU can handle them easily (each one needs a single 6-pin PCIe power cable).

I installed a default no-frills Proxmox PVE 9 on a 500 GB SSD which has enough space for two VM images that need to contain a full Rocky Linux 9 installation with a graphical desktop environment. Any smarter setups in terms of storage are beyond the scope of my tutorial.

How To

First, Proxmox itself needs to be modified a bit. It must not load any graphics drivers during its boot process as this would prevent the graphics cards from being passed through to a VM. It’s probably best to install all available system updates first and reboot in case of a kernel update. Check that you have both SSH access and that the Proxmox web GUI is working fine.

Create /etc/modprobe.d/blacklist.conf and enter these lines to prevent Proxmox from using these kernel modules:

blacklist nouveau
blacklist nvidia

Modify the GRUB_CMDLINE_LINUX_DEFAULT line /etc/default/grub (you can duplicate it and add a # in front of it to preserve the original line in case you want to undo this later):

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

Run update-grub afterwards!

IOMMU is Intel’s name for the feature that allows passing devices to VMs. It can only work if it is supported and enabled, which was true for an old machine like my HP Z820 workstation so it’s probably supported on all modern hardware (it’s called VT-d inside the HP BIOS).

Create /etc/modules-load.d/vfio.conf and add these lines to load some more required modules.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Once this is done, you can reboot your Proxmox PVE and you’re done with meddling on the command-line. If you have a monitor connected it should stay blank and not show the Proxmox terminal login anymore.

Note:

The article I’ve mostly followed myself contains more information about IOMMU groups. It was not necessary to deal with this in my case, probably due to the good server-grade hardware I had.

VM Creation

Create a new Virtual Machine. The required settings for GPU passthrough are as follows:

• Graphic Card: Default
• Machine: q35
• BIOS: OVMF (UEFI)
• Add EFI Disk checked
• Pre-Enroll keys unchecked (important if you don’t want secure boot inside the VM which forces you to do some extra things to get the nvidia driver working. See this forum post.)
• SCSI disks, vCPUs and memory as you like. I’ve assigned half of my cores to each VM and a bit less than half the RAM each.

Once the VM is up and running you need to pass through the GPUs and optionally some peripherals. I’ve installed Rocky Linux 9 in it before doing the next steps but you can probably also continue right away. Just remember that the virtual browser-based console will stop working once you have passed the graphics card to the VM. If you are not in front of the host PC you should install Linux first.

Make sure the VM is stopped and go to the VM’s Hardware section. Click Add → PCI device. Select Raw device and pick your NVIDIA card from the list. I had 2 identical GPUs in my case and chose one for the first VM and the other for the 2nd VM. I’ve also checked the boxes for Primary GPU, All Functions, ROM-Bar and PCI Express. There’s probably also an audio device provided by the NVIDIA card. I didn’t pass it through as I didn’t need it.

To pass through a local USB keyboard and mouse connect the peripherals and click Add → USB Device. You should be able to pick the devices based on manufacturer or pipe through the raw USB port. The latter, however, was a problem for me as a reboot of Proxmox seemed to have jumbled the port IDs around. The device manufacturer’s IDs were safe but if you ever swap your keyboard it will need to be added to the VM again.

Inside the VM you should now see the NVIDIA device listed if you type “lspci”. You just need to install the drivers now which are available from the 3rd party repositories ELRepo (kmod-nvidia) and RPM Fusion (akmod-nvidia). After a reboot of the VM you can check nvidia-smi or btop to confirm that the GPU is actually working.

Mastodon

My blog hasn’t been updated for a long long time and will be on hiatus indefinitely.

You can find with me on Mastodon as @compfu@mograph.social instead (https://mograph.social/@compfu) where I post about things that are on my mind and boost motion graphics and VFX related clips I stumble upon.

Yawn In The Shell

The life-action Ghost In The Shell (GITS) movie is pretty forgettable. Sure, they mostly nailed the visuals. But it’s 2017 and movies shouldn’t get points anymore for just “looking good”. All the tentpole movies do.

VFX on par, story bland

Of course the original anime was heavy on amazing visuals. But it was the 1990s and back then this kind of stuff had never been seen before. GITS (1995) even slowed down at one point, showing scenes of futuristic city life that didn’t advance the story but were simply jaw-dropping. Somebody actually hand-animated all of this! Fast forward to 2017 and rendering shot after shot of a futuristic Tokyo really isn’t all that impressive anymore. Yeah, you can have the industry circle-jerk about how innovative their VFX were just as they do after every effects-laden movie. But come on… yawn!

The GITS movie feels like it came late to a party. The original anime, while certainly not inventing gritty cyberpunk, influenced cinema and genres for years. In a way you could call the Matrix movies the first life-action adaptations of GITS. Now, in 2017, everything in GITS seems like it has been done before. So did the story at least bring something new to the table?

Unfortunately, the story of GITS feels pretty bland. To say that it “explores” its central issues like what it means to be human in an age where people enhance their physical bodies with computer interfaces is a gross overstatement. Most people in the movie’s universe don’t care and get along well with their cyber-enhancements. We are told that a teenage hot-head once said that technology needed to be destroyed. That’s about it.

Of course the GITS anime – when viewed on its own – is also heavy on pseudo-intellectual mumbo-jumbo. Its sequel (Ghost in the Shell Innocence) even more so. But GITS has spawned a couple of spin-offs (the “Stand Alone Complex” series for example) that fills the franchise’s futuristic world with details and the special forces team called “Section 9” with life (and cute robots). The GITS movie doesn’t seem to pull anything from this.

Tropes, tropes, tropes

GITS really feels dumbed-down. In the movie’s first couple of minutes somebody explains to you that “Ghost” means mind and “Shell” means body. The boss of the shady high-tech robotics company straight-out declares the newly-built Scarlet Johansson cyborg a “weapon” in a voice that says “I’m the boss of a shady high-tech robotics company and will definitely be the antagonist later when she finds out what we did to her.”

The movie then continues to visit all the lame action-movie tropes like the bad guy switching an automatic weapon system to “manual mode” so he can miss his target more easily and ultimately get defeated. A tank does the cliché clicking/snarling sound from Predator that’s usually used for organic critters approaching a victim.

The movie then devolves into a cringe-worthy scene between Scarlet Johansson (“Major”) and a former friend and/or lover who also got turned into a cyborg that feels like a prom queen politely refusing the awkward advances of a pimple-faced nerd.

The Casting Controversy

But wait, wasn’t there some kind of controversy surrounding the movie? Something about Scarlet Johansson playing a Japanese character? (I’ve talked about this issue in a German post about Cloud Atlas). Well… the Japanese audience didn’t seem to care (the movie flopped in the US but made some bucks in Asia) so who are we to judge.

The casting is bad for a different reason. It actually harms the story as well as suspension of disbelief. At one point, “Major” is confused about living in a robot body. She visits a hooker who tells her that getting touched by her feels different. “What are you?” the hooker asks with a confused face. And rightly so. Even in the world of GITS the technology of putting somebody’s actual brain into a fully cybernetic bodies is bleeding-edge. Johansson is the first of her kind.

“Major” then finds out her past and meets her mom – an old Japanese lady. In an already unbelievable scene that woman immediately invites Johansson into her home and starts an exposition dump about her deceased daughter – Major’s former identity – because Johansson “reminds her of her daughter”. She talks English with a noticeable accent since apparently she recognises Johansson as non-Japanese. But when she realises that Johansson is in fact her former daughter she doesn’t mention a single word about that fact. No “you don’t look anything like her anymore” or “what are you” or “it feels strange hugging you”.

The whole boring movie could have been taken up a notch by simply having the mother acknowledge that her daughter now has a cybernetic, western-looking body.

But that didn’t occur to anybody who had a say in the creation of this movie. And that says everything you need to know about it: GITS 2017 is an effort at making some bucks by bringing 20-year old anime visuals to a life-action movie. Nothing more.

6 out of 10 – the “don’t spend money on this even if you are a fan” level

Data Loss

My web host experienced data loss while repairing a crashed RAID. So this means that my blog is reset to last March. I’ve grabbed a few posts from google’s cache but didn’t have the time yet to transfer them back into WordPress.

Update, one year later: I’ve finally found time to restore a couple of posts from 2015. See below.

Adding Layers and Channels in Nuke

If you need to create new layers in Nuke via Python, here’s the proper syntax:

nuke.tcl("add_layer", "newLayer newLayer.red newLayer.green newLayer.blue")

That was a bit hard to find in the docs as well as online and there’s some weird behavior (at least in Nuke 8) when this is called in a script vs typing it into the scripting console.

For example, the tcl input (“x” hotkey) requires you to enter the command as “add_layer {myLayer red green blue}” and it reminds you of the curly braces syntax in case you do it incorrectly. The .tcl method in Python, however, mustn’t contain curly braces.

Moreover, nuke.tcl requires you to spell out the rgb channels in dot format. If you just use “red green blue” Nuke will create a layer called “other” for some weird reason and probably crash sooner or later. It only does this in Python scripts though. Not in the scripting console.

You might have noticed that I’m calling a TCL command via Python. There’s also a native way: nuke.Layer(). Just like its tcl counterpart it requires you to spell out the channel names – even if the official docs say otherwise:

nuke.Layer("newLayer", ["newLayer.red", "newLayer.green", "newLayer.blue"])

Animated Homage to Miyazaki

Ah the French and their top-notch animation colleges… Here’s a Créapole ESDI student’s diploma project – an exceptionally well-animated homage to Studio Ghibli and Hayao Miyazaki.

httpv://vimeo.com/124313553

References to famous anime movies everwhere! Slimy gooey monsters from Princess Mononoke or Chihiro, giant airborne insects and rural people with funny masks from Nausicaä…

Incidentally, this short film makes you aware that Miyazaki’s work (or anime in general) is a constant rehashing of familiar tropes and the same characters. Take the “wise old bearded guy” from the vimeo still frame above. Or the burly light-hearted sidekick. The weird/nerdy heroine. The rural setting with windmills and nature as a god/goddess symbolized by a giant tree. It makes you think you can pull off your own Ghibli movie by just reassembling Miyazaki tropes.

Which of course isn’t true, and even if there are many many tropes in anime, this isn’t necessarily a bad thing. It’s probably even the “magic formula” that gives the audience more of the same that they’re craving for.

Well, I went off on a tangent there. The short film by Gwenn Germain makes you wish it was a trailer for an actual movie. I’d watch it!

There’s more at The Creator’s Project. Link found via pixelsham.

Trolling International Relations via VFX

This is the story of a great hoax no matter what the true story is: after the German media has made a fuss about a 2013 video of the Greek treasury secretary Varoufakis flipping the finger, a small TV show has claimed that the video is fake and that they have doctored it (enable English subtitles). At the end of my post I’ll demonstrate why I think they are making this up.

 

A bit of context:

German’s biggest yellow press newspaper has been spewing tirades and stirring hatred towards Greece for years now. By now, viewing Greeks as lazy thieves of hard-earned German monies is basically ingrained in our national narrative of Europe; as shameful as it is.

The team of a small late night show claims to have trolled the media by inserting the obscene gesture and seeding the video to Youtube. But did they? They might have doctored the video the other way around for their show to remove the gesture of Varoufakis and make a point about the current German mindset.

From a technical point of view, either theory is plausible. I’ve done VFX to replace body parts myself and that section of the Nuke comp that they’re showing makes some sense (except for the motion blurred grid they used as a making-of effect). It would certainly possible to make Varoufakis raise his middle finger for a few frames.

But it would be an equal amount of work to remove the gesture: retime, freeze and matchmove his torso to postpone the upwards motion of his arm by a few frames, then transition back to the original footage.

 

To those who are amazed by what you can do with technology these days: that stuff has not only been possible for 20+ years (see Jurassic Park’s stunt double face swap which was still a bit rough around the edges). It has been done seamlessly on all kinds of motion pictures for quite a while.

Analysis:

So can we find out which clip is real just by comparing both versions? Yes. I think the finger is real (side note: and I couldn’t care less…) and here’s why:

Let’s start with the obvious. The idea that you would fake this by employing an actor who looks like Varoufakis and making him wear a green full-body suit except for his hand and collar is ridiculous.

It’s playing with the general public’s idea of how green-screen photography works and it has been spoofed a lot of times already, for example by MADtv years ago and anotherGerman TV show more recently.

But let’s look at the hard facts of the footage at hand. Here’s a frame-by-frame comparison of both clips next to each other. No matter which version is real, they did a great job.

 

To fake the left version (the flipped finger), they would have had to replace the right side of his torso to get rid of the lowered arm and during (what would in this case be) the arm’s real upwards motion. That’s much more than just pasting a new hand on top of Varoufakis as they make us believe during the making-of. That’s a great comp job (if it were true).

By the way: the nature of the shirt he’s wearing might make doctoring this footage easier. Wrinkles pop up and vanish due to the tiniest motions of his body so you can hide masks and transitions better than you might think.

If the right version – without the middle finger – was fake, they would have had to freeze, replace and/or morph a large part of Varoufaki’s torso and parts of his collar to get rid of the raised hand for an extended period of time, all while Varoufaki’s face keeps moving/talking. That’s great comp job in this case as well.

But here’s the real deal: They would also have to mess with the other hand that is holding the microphone! And they did.

Here’s how to spot the fake:

You can see in the left shot (the one that I think is genuine) that there’s a shadow running over the mic while Varoufakis raises his hand. On the left side (the one that I think is fake) the whole hand and microphone area is excibiting a weird wobbling motion that starts as soon as they had to retime/freeze his torso and stops as soon as he lowers his arm.

To me, this is a sign of doctoring. A side effect of the warping they had to do to match a non-shadowed microphone on top of the real footage.

And then there’s the artifact which is the ultimate clue in my opinion.

Right after his hand has moved down, they transition from their fake clean plate back to the real microphone. But the cable has been twisted ever so slightly while Varoufakis was flipping the finger that you can see the blending of two versions of the cable.

Verdict:

The finger is real. The VFX artist(s) did a great job to undo this gesture and the show is (successfully) trolling the media by making them think they fell for a doctored video. ZDF neo and its host Jan Böhmermann are successful in making the general public question the way they get enraged by unimportant videos that were taken out of context.

Further reading: The Ems Dispatch, a purposefully edited document that played a part in starting the Franco-Prussian war in 1870.