You can nest the outer formatting inside the inner formatting by using double-hash on the outside and single-hash on the inside.
The following worked for me. I put it into separate lines for improved readability, but it still works when you put everything in one line.
This is normal sized text.
[.small]##This is small text.
[.underline]#However this part must be small and underlined#
##
This utilizes the constrained/unconstrained formatting as described in the user manual:
https://asciidoctor.org/docs/user-manual/#constrained-quotes
Regards, Alexander