So I went forward with that plan. Unfortunately, it didn't work out the way I wanted. I just couldn't get the normal map to look good enough. Two days of work went straight into the trash.
So, a little dejected, I decided to take a more complicated approach to the material. I would model high polygon stones and bake the normals on to a plane. This workflow would take far longer, and had it's fair share of complications, not to mention I wasn't too excited about modeling high polygon rocks since I had recently modeled a ridiculous number of stone blocks.
Here are the results thus far rendered in the Unreal Engine.
The material uses the baked normals map and also a greyscale height map to create the parallax bump effect. A color and roughness map are also included.
In the end, the colors may change a little, but at this point, I'm pretty happy with the results. I plan on making a dirt material that has the same consistency as the dirt between the rocks. Then, I'll blend the two materials together using a vertex blend or some other fancy mapping. This will add some variety and a really cool worn out look.
So what's next, you ask? The ceiling material. That should be a relatively easy endeavor, and then I'll put a full testing room together!