OK I have more details on how to reproduce:
Since this is a macOS installer, I only offer it to older versions via the fileset requirements. So when I made this fileset and set the requirements, it gets offered to a computer running macOS 12. I go ahead and install the app from the kiosk, it runs its script and essentially updates the computer to macOS 13.
Since the fileset is installed, its now ignoring the requirements. My guess is its figuring if its installed already then there’s no need to check for requirements afterwards. However this is how I’ve deployed previously macOS 11 and 12, and once the macOS was updated the fileset requirements would kick in and the installer would disappear from the kiosk. This is no longer happening.