I have finally solved the back-face culling problem using my own formula. Press space to switch between cube and octahedron.
I found the angle between the surface normal and the z-axis for three points where their projections are collinear, and applied it to multiple sets of points for one face after rotation.