This is the solution:

Using single-quotes instead of double-quotes
Escape the ampersand & to \&
Not escaping the exclamation mark !

e.g.

docker run -i -t -rm \
-e 'LDAP_FILTER=(\&(objectCategory=person)(objectClass=user)' \
-e 'LDAP_PASS=Secret!Password' \
user-prefix/container-name

en_USEnglish