Max is found when all of the "extra" players are all on the same team. You have the solution to that.
Min is found when all of the players are spread as evenly as possible across all of the teams. So, if the number of players is evenly divisible by the number of teams, all teams will have n/m players. If the number is not evenly divisible, the remainder (n % m) is the number of teams that have 1 additional player.
After that, just use pairs() to calculate the number of pairs for each group size and then multiply and add to get your answer.