The closer a game resembles film, the more casual players are scared. Conversely, the less scripted a game is, the more the core players are scared. Third-person, tightly scripted events are scarier to casual players than to core gamers, while first-person, generative events are scarier to experienced players.
I think what happens here is the following:
Casual gamers are expecting the game to drive the story and therefor the scares with them. They expect to be delivered little pieces of fear and the game to focus its whole attention on them. In short: I accuse them of wanting to be spoonfed with terror.
Core gamers are likely to explore more on their own without being given incentives. They explore for the sake of discovering the things the developers want them to see. As can be seen in the rising excitement of the core gamers amongst the study's test subjects the tension doesn't sink when exploring.
My impression is that a gaming experience which relies heavily and nearly exclusively on scripted sequences (you might call it a on-rails-experience) don't excite core gaming audiences as much as more freely explorable parts tend to do. This is caused by a motivation to see what might lurk in the shadows (pun intended) instead of heaving something jump at you from the vents.
That being said one might argue that all gamers found Dead Space 2 the scariest game tested. Considering that I personally wouldn't count Alan Wake as horror game and there've been many voices exclaiming that Resident Evil 5 was way more of an action game than a scary horror game… no, wait, I'm rambling. What I wanted to point out is that Dead Space 2 used techniques that the some of the other games seemed to lack or not use effectively, like strange noises to suggest impeding danger. To conclude this little article I'll just quote another short paragraph of their findings:
Actual combat is not as scary as the implied threat of combat. The biggest cares result from moments devoid of any physical combat; instances in which players anticipate or fear they are about to fight, but do not actually end up doing so.