I just realised after lots of twist and turns that when scheduling a powershell job, the Start in value will not work if I just put the Program as "powershell.exe". If "Run with highest privileges" is checked, the working directory of your script will just be "C:\Windows\System32" regardless of what value you put in Start in. Just like it would if you launch powershell as administrator from the start menu. I haven't tried this, but I guess if it's not checked, the working directory should be %HOMEDRIVE%%HOMEPATH%
To solve the problem, just change the Program value to the full path. In my case, this is "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe". Quite a handful to type, but it's the only way I managed to get it to work so far. If there are any better suggestion, I'd be glad to hear.