> for(int j =1; j<= i * i; j++)
This is O(N2), because you have i*i (which is O(N)).
By themselves, the first two loops are O(N3).
> for(int k = 0; k<j; k++)
This is O(N2), because j.
But what fraction of if (j % i == 0) is actually true for some asymptotic N ?
If it's relatively rare, perhaps it's just O(N).
Perhaps it's very rare, perhaps even log(N).
This is a nice problem. Here is my 10 cent take on it.
outer loop is n, of course (not my N loop, ignore that).
the next loop is the sum of squares times:
(N * (N + 1) * (2N + 1)) / 6 (this is O(n^3).. its ~ (n*n*n)/3 )
That is the combination of the first 2 loops.
then the weird one.
the k loop happens about n*n/2 times. each time it does, it adds i*i sums.
I think where we end up is O(n^4). Its significantly less than that due to additional terms that are ignored in big-O. I don't know how to deal with the changing i term here (if you wanted it exact).
My final approximation was (N^4)/6 .