You array is (1000000 * sizeof(long double**)) + (1000000 * sizeof(long double *)) + (5 * sizeof(long double)) bytes large. Don't think you have the memory for that. What is it, 20 tB?
On the last level you're creating over a million small arrays. Since successive calls to malloc() doesn't allocate contiguous memory blocks, it's possible (although IMO unlikely) that this overhead is causing problems.
Does newlongdouble[h*h*k] fail as well?