If you want to validate input XML files, then you can either validate with an XSD or via a rule based (schematron) validation.
If you want to use a rule based validation, then you first need to upload a schematron file (.sch) in the Validation rules folder/tile. The uploaded schemation file can then later be referenced in a rule based validation.
XSD validations refers to XSD files placed in the transforms library.