I compiled a mex-file from a C++ code which use GMP and some uint64_t (I'm on a winXP 32 bits).
This C++ code works perfectly in C++.
But I noticed it just hangs when I run the mex-file in Matlab for some inputs (it just hangs, I have to kill the process).
What I don't understand, it's that if I run it using the -nojvm Matlab mode, it works perfectly. So I guess it's related to the Java Virtual Machine. I already tried to increase the heap limit to 1024mo, but it didn't fixed the problem.
When I manually follow the Matlab run when it hangs, the C++ instructions seems to stop without explanation (not an infinite loop or something).
Each time I increased the java heap limit, the Matlab process memory seemed to increase too. But if I run the exact same code in a classic main.cpp C++ file, it doesn't seem to use as memory.