I believe that in the case of operators having the same priority ranking, it is simply read from left to right. Since * and / have the same priority, it will calculate the n*(3*n-1) before it gets to the /2, and because it overflows (max value of signed long being 2^31 -1), the division happens to the negative figure.
Someone feel free to correct me though.
edit: I just tested it with the same result, and then I did the same with the /2 earlier in the formula and got 2 positive numbers.
A signedlong variable can store values up to 2147483647.
When you call f(26756), according to the order of operations, the compiler performs three multiplications before it divides in n * (3 * n - 1) / 2. The problem is that n * (3 * n - 1) > 2147483647; this causes the overflow.
If you rewrite the operation like this, it will work: n * (n - 1) / 2 * 3.