This can definitely be further improved, as there are Citra components which are also built unnecessarily, but that would require invasive CMake changes which are out of scope for this PR.