I recently removed wallpaper on a wall to find mortar leftover form removing brick veneer. how do I remove it?

I am not too keen on putting back a wallpaper. I do not mind a textured finish to the wall. i have heard nightmare stories of using muriatic solution. Is there a simpler way to finish this wall other than taking down the drywall?