It is very important but quite difficult to test the security of an operating system. In this paper, some essential problems about security testing of an operating system are discussed, including conception and extension of security testing, feasibility and technical scheme for automated security testing of an operating system, security of system calls, testing sequence for system calls, and etc. Thereafter, a prototype system (i.e. a series of testing tools) for automated security testing of system calls is designed and implemented based on Fedora 9 and Linux kernel 2.6.25-14.fc9.i686, which is made up of control module, objects setup module, standard test module, special test module and test configuration database for each system call. Furthermore, test cases as well as test results for systems calls such as creat, access and etc are discussed and analyzed. Finally, the research work in this paper is summarized while further research directions are pointed out.