I’m not sure I quite understand the issue Steve has with things updating when adding new cards.
Sure, updates during a single benchmark series is a problem but what is the issue with the system being updated for the next benchmarks?
Proton/amdgpu/Mesa receiving updates is no different than installing newer “Game Ready” drivers when a new GPU comes out.
I assume they don’t go out of their way to install older (potentially incompatible) drivers on Windows just so they can compare two separate benchmarks.
Besides that, after disabling Flatpak and rpm-ostree updates in Bazzite, the only remaining variable is Proton. Which should be easily fixed by manually copying a fixed Proton version to their compatibility tools and using that.
Without re-testing their entire suite of cards for every new card review (which is cost prohibitive), performance changing from updates would make the comparisons between cards less useful, as it cannot be determined if the newer card being tested is better or worse purely on the merits of the hardware itself, since newer software may be artificially making it look better or worse than the tested cards that came before, and thus the actual integrity and usefulness of the testing comes into question.
They are trying to assemble a like-for-like dataset that doesn’t require their entire catalog of cards to be regularly retested to ensure that it remains like-for-like. Keeping all the software the same across tests ensures that they can add new data piecemeal and still retain an apples-to-apples comparison.
AFAIK, It’s not an issue of automated testing, and I don’t believe they re-test all their cards on Windows with every new review either. Instead, they maintain the same versions of software on Windows as well until enough time has passed and enough updates have piled up that they do finally re-test everything with new games to create a new dataset to compare against. They’re trying to do the same methodology on Linux.
Instead, they maintain the same versions of software on Windows as well until enough time has passed and enough updates have piled up that they do finally re-test everything
I’m not that involved with their testing procedure but doesn’t that put newer cards at a disadvantage?
They lack any sort of driver optimization if the release drivers are never installed.
That’s a good point. I went back to the video to rewatch it, and turns out I totally missed where they said they only freeze things during a testing phase, then unfreeze it after they’re done and allow updates to commence as normal.
They mentioned that due to Linux receiving more frequent updates often with meaningful performance improvements, they’ll have to throw away older data and re-test more often on Linux, as Windows doesn’t really change much in performance between updates. So I would guess that they would use release drivers with new cards, and likely would only re-test their entire suite if the release driver also gave a big performance boost on older cards.
https://www.youtube.com/watch?v=ovOx4_8ajZ8&t=2746
I’m not sure I quite understand the issue Steve has with things updating when adding new cards.
Sure, updates during a single benchmark series is a problem but what is the issue with the system being updated for the next benchmarks?
Proton/amdgpu/Mesa receiving updates is no different than installing newer “Game Ready” drivers when a new GPU comes out.
I assume they don’t go out of their way to install older (potentially incompatible) drivers on Windows just so they can compare two separate benchmarks.
Besides that, after disabling Flatpak and rpm-ostree updates in Bazzite, the only remaining variable is Proton. Which should be easily fixed by manually copying a fixed Proton version to their compatibility tools and using that.
Without re-testing their entire suite of cards for every new card review (which is cost prohibitive), performance changing from updates would make the comparisons between cards less useful, as it cannot be determined if the newer card being tested is better or worse purely on the merits of the hardware itself, since newer software may be artificially making it look better or worse than the tested cards that came before, and thus the actual integrity and usefulness of the testing comes into question.
They are trying to assemble a like-for-like dataset that doesn’t require their entire catalog of cards to be regularly retested to ensure that it remains like-for-like. Keeping all the software the same across tests ensures that they can add new data piecemeal and still retain an apples-to-apples comparison.
That makes sense.
So the best option seems to be to note updates for newer cards down until the automated testing can be done on Linux as well.
AFAIK, It’s not an issue of automated testing, and I don’t believe they re-test all their cards on Windows with every new review either. Instead, they maintain the same versions of software on Windows as well until enough time has passed and enough updates have piled up that they do finally re-test everything with new games to create a new dataset to compare against. They’re trying to do the same methodology on Linux.
I’m not that involved with their testing procedure but doesn’t that put newer cards at a disadvantage?
They lack any sort of driver optimization if the release drivers are never installed.
That’s a good point. I went back to the video to rewatch it, and turns out I totally missed where they said they only freeze things during a testing phase, then unfreeze it after they’re done and allow updates to commence as normal.
They mentioned that due to Linux receiving more frequent updates often with meaningful performance improvements, they’ll have to throw away older data and re-test more often on Linux, as Windows doesn’t really change much in performance between updates. So I would guess that they would use release drivers with new cards, and likely would only re-test their entire suite if the release driver also gave a big performance boost on older cards.