With these settings, the overlay is being scaled with bilinear filter, which is blurring some of the scanlines and causing the moire effect you were describing. IMO this can be very distracting, particularly with vertically scrolling games.
I’ve detailed a way to get pixel perfect scaling at 1080p while cropping both the letterboxing (the black bars) and the overscan, so that the picture fills the vertical area of the screen and the scanlines line up perfectly without scaling artifacts, cf., “how to get perfect video scaling” and “how to get scanlines.”
There are a couple incorrect assumptions you’re making, one of which is that most systems put out 224 horizontal lines of resolution, which is not the case. If you’re losing important score/credit info, as you said you were at 5x scale, it’s likely because you are using 224 as the internal resolution instead of 240. What gets cropped at 5x scale corresponds exactly to the overscan area on an NTSC CRT TV. The remaining area is referred to as the “safe zone,” and game developers would always make sure that important game graphics were displayed within this zone (with a few rare exceptions), and they frequently made use of the overscan area to store information that the player wasn’t supposed to see (usually, to compensate for the variability in how CRTs were calibrated).
Also, regarding opacity, IMO you’re better off setting your display’s “backlight” setting higher to compensate for the loss of brightness. (This is not the same as the “brightness” setting). For example, with 4x scanlines at 100%, you would lose 50% light output, so you compensate for this by increasing the backlight by the same amount (eg., if your backlight is at 50%, you would turn it up to 100%).
Also, regarding the dotmask shader, it will cause a drop in frame rate and input lag, but it is slight. I’m sensitive to these things, so it ruins it for me, personally.
With this said, could you please correct the misinformation contained in your post regarding the horizontal resolution of console systems? This kind of information can quickly and easily spread once posted, leading to more confusion and problems, and it works against the efforts of those who are working to preserve authenticity and accuracy. Thank you. :)
The correct horizontal resolution is 240 for NES, SNES, Genesis, PS1, N64, PC Engine. This can be readily corroborated by using Google and searching “internal resolution (system name)”
However, there are individual core differences and bugs where the internal resolution isn’t right, Picodrive being a notable example. So the correct resolution for a system may not be the “correct” resolution for an emulator. I’ve saved you the work by providing a list of resolutions to use in “how to get perfect scaling.”
I haven’t tested Neo Geo, but I think it varied, with the most common being 264 with overscan (240 without)