Last tested: Aug 26, 2020
This can be accomplished using the SP Entity/IdP Audience (documentation) setting in the Admin > SAML settings. You can assign this field to only specific people within the IdP to be allowed to log in to Looker. If a user attempts to log in that does not have this value assigned, the authentication with fail and the user will not be able to access Looker. This will also prevent them from creating an account until they are assigned this value and can successfully authentication.
Usually this is done through groups on the IdP side but different providers have different terminology. The Audience field is how they specify the group name in the Looker settings.