For Quake1 he wrote a little script language around each step of setting up the various modes. With that, it was much easier to navigate the maze of “Step A and C both failed, that means it’s a Type 3 device and Step D will crash so, we should skip to the Branch 3 Path and hope for success at a lower resolution.”
That was not just true for VGA but for many of the ...interesting... cards which were put on the market 'back in the day'. I had one of those, a 'Diamond Brand' multi-IO + Hercules + memory card without documentation or drivers I happened upon at one of the then-ubiquitous hardware dump meets - it was the 'PC Dumpdag' in Amsterdam for those who remember that phenomenon - with which I spiced up my otherwise somewhat boring Bondwell BW 32. While working on a TSR (as in 'background routine') which would make it possible to automate data entry into some school management software package I ended up flipping a bit on the 6845 display controller on the thing upon which the vertical resolution suddenly doubled. Instead of 80 columns 25 lines it now displayed a somewhat compressed and initially messed up 80x50. Experimenting further I got it up to 80x70 and 132x70. I made another TSR to enable mode switching and used this quite a lot when working in Wordperfect and the like. It also did double the normal (720x348) graphics resolution which I got to work in a few programs but which was less useable than the text modes. I tested this program on many other boards but never found another one where it worked, alas.
Also, I remember that with some tools that come on a floppy disk with the card, I could output a 1024x768 graphics scree, but with a horrible blinking... I think that I was putting my multi-sync monitor at the limit.
* Graphics card RAMDAC bandwidth was a limiting factor for higher resolutions x higher refresh rates, too. And video memory limited higher desktop resolutions x higher color depths for years - really until 3D accelerators were coming of age.