A panel of judges said the plaintiffs ‘lack any direct evidence of racial motivation’ in their legal challenge.