I can easily imagine a hacker getting physical access: you're a college student who shares a dorm with another student.
You're not thinking big enough. Try a janitor cleaning the room of the CEO of a fortune 500 corporation.
A lot of times, physical access is considered secure if the machine's hardware is locked down (hacker's can't open the case without making it obvious it's been tampered with, hacker can't take the computer without someone noticing it's missing), and the software is blocked with a secure login. Cortana being active when a user is not logged in is a gaping security hole.
The proper fix for this is to neuter Cortana so the only thing it can do if the account is locked or logged out, is to help you login. But that probably gets in the way of Microsoft's data collection. So they force us to live with the gaping security hole. I guess we need to add a locked switch physically disconnecting the microphone cable inside the PC alongside the piece of electrical tape over the webcam.