So there is a article written by Microsoft stating for the most part that you should not use Windows as a reliable NTP server.
The introduction: the support boundary to configure the Windows Time (W32Time) service for high-accuracy environments. High-accuracy environments include exchange trading houses and high-precision manufacturing companies for whose success critically accurate time measurement is required.
Later stated: We do not guarantee and we do not support the accuracy of the W32Time service between nodes on a network. The W32Time service is not a full-featured NTP solution that meets time-sensitive application needs. The W32Time service cannot reliably maintain sync time to the range of one to two seconds. Such tolerances are outside the design specification of the W32Time service.
Well it seems like its time to switch over to Linux for my NTP needs.
Ref: kb939322