You are asking too broad of a question. I will try my best to answer your question.
heuristic development in an AV solution, requires you to understand how Malwares behave in general and API's they exploit in order to perform their malicious deeds. In quick words, heuristics in a AV solution is basically building a picture of how the process works.
To follow this and provide more detail, check for common Malware behaviour:
- Injection Activity
- Network Activity
- Registry Activity
- File Activity
Further tracking includes:
- MBR\VBR monitoring
- System Driver Installation
To expand\elaborate ever further, to track these behaviors it requires you to intercept key
API's used by Malware via placing hooks on common API's.
I have included few common API's Malwares use, thus hooking them would be ideal:
11. NtCreateThreadEx (Vista+)
In addition Windows contains useful functions which can be called to build\acquire vital pieces of information, which can help track Malwares on the System.
1. CmRegisterCallback (Receive notifications on virtually all registry events.)
2. PsSetCreateProcessNotifyRoutine (For new/terminating process notification)
To make, this job easier and more effective - I advise you place hook on KiFastSystemCall (on x86 machines only).
As for x64 machines, you can either perform\place hook on X86SwitchTo64BitMode or, if you wish to hook even lower hook Wow64SystemServicesEx (available on x86 process only, but using x64 DLL loading), this would make it almost impossible to bypass.
Moreover, those functions has access to all NT system calls, therefore you can hook the entire userland by placing hook on 1 functions.
Lastly, I would recommend monitoring the MBR\VBR as if a Malware subverts the MBR\VBR, everything is untrusted.
To hook KiFastSystemCall - visit my blog, to get complete source code to hook KiFastSystemCall:
As for the other functions, I am going to be posting them soon.