The remaining small shack also had some serious issues with rendering the interior view properly when someone was inside. The roof just flickered on and off and anyone entering appeared to sink into the building and then pop back up.
EDIT: I figured out what was causing this. The invisible wall bug happened in the middle of my outpost and all buildings on one side disappeared. Closing and restarting the game solved this issue. Sort of. All the buildings now have the flickering roof glitch and sometimes my men just walk straight through them, which makes them mostly useless.
Additional info: Reloading the game without first closing it causes all the buildings to disappear again and the invisible wall resurfaces. Reloading the game after closing it and placing my men on the opposite side of the invisible wall causes all buildings on the side they are not on to disappear and all buildings on the side they are on to reappear. The buildings that have appeared all suffer from the flickering ceiling/ghosting issue and are basically unusable.