Modify PowerShell command used to run scripts so that errors occurring outside the script itself can be caught

History here: https://community.syncromsp.com/t/scripts-showing-as-successful-even-when-they-fail-because-script-execution-isnt-allowed/8206

In short: The current command that Syncro uses to run scripts leaves open the possibility of failure due to PowerShell execution policy or other issues preventing the script from running, while incorrectly marking the script run as successful. This could be fixed pretty easily by modifying the command.

— Overly-lengthy explanation of the problem: —

What Syncro does with scripts is package them as a PS1 file, download them to the asset, and then run a command to launch the script. Here’s that command (the name of the script is randomly-generated):

"C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe" -Sta -ExecutionPolicy Unrestricted -Command "& {C:\ProgramData\Syncro\bin\a575560c-0b80-42cb-b0bd-0349d86e110e.ps1; exit $LASTEXITCODE}"
  1. “C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe” - Launching PowerShell
  2. -Sta -ExecutionPolicy Unrestricted - Declaring STA mode and attempting to set execution policy to unrestricted.
  3. -Command "& {C:\ProgramData\Syncro\bin\a575560c-0b80-42cb-b0bd-0349d86e110e.ps1 - executing the downloaded script.
  4. ; exit $LASTEXITCODE}" - Taking the exit code from the script (if one was given) and exiting the PowerShell session with that same code.

The problem really comes in the last step. If the script declares an exit code the system variable $LASTEXITCODE is automatically set to that value, whatever it may be. That variable is then used as the exit code for the PowerShell session that Syncro first spun up. When that happens, the Syncro console picks up the code and uses it to determine and display in the web UI whether the script run was a Success or Failure.

If the script did not exit with a code, no code can be used to populate the system variable, and therefore the parent PowerShell session exits without a code. The Syncro console has to make an assumption of Failure or Success in this case, and they programmed it to presume Success. This is fine in my opinion. Presuming failure would force people to design scripts in ways they may normally not, and only fixes a small portion of the problem anyway.

The real issue happens either:

  1. When something prevents the script from running in the first place - execution policy doesn’t allow it, AV solution deletes the script before it can run, etc. - so no exit code can be passed to $LASTEXITCODE, even if the script was designed to do so. A script can’t exit if it never starts.
  2. When a script runs into an unhandled exception and terminates before passing an exit code.

— Potential solution: —

Add the following to the command after the PS1 file is launched and before the exit statement:

if(($? -eq $False) and !($LASTEXITCODE)){$LASTEXITCODE = 1}

$? is a boolean system variable which is True if the last command executed was successful and False if it was not. This applies to basically any command (including a command to display the variable itself), and any possible failure of that command. See this example screenshot from a system on which I’ve set execution policy to restricted, so scripts are not allowed to run:

Here’s another example. In this one, test.ps1 is a script that contains only a “throw” statement, and test2.ps1 is a script in which I intentionally left out a curly brace.

ps2

So $? will display False if either the script itself fails with an unhandled exception or if the command to run the script in the first place fails for whatever reason. We can use this in an inline IF statement before the exit by checking if $? equals “False” and - because we wouldn’t want to override an exit code passed by the script if there was one - checking for an exit code. If $? is False and $LASTEXITCODE is blank, we set $LASTEXITCODE to “1” so that it can then be passed to the exit statement. It would look like this:

"C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe" -Sta -ExecutionPolicy Unrestricted -Command "& {C:\ProgramData\Syncro\bin\a575560c-0b80-42cb-b0bd-0349d86e110e.ps1; if(($? -eq $False) and !($LASTEXITCODE)){$LASTEXITCODE = 1}; exit $LASTEXITCODE}"

Thus we can inform the Syncro console and web UI (and ourselves) of errors to which all parties currently remain blissfully unaware.

4 Likes

Another example of this just popped up today. Our AV solution sniped the module.psm1 file which causes scripts to fail, but the Syncro console still shows Success. The full output of the script run looks like this:

error> Import-Module : The specified module ‘C:\ProgramData\Syncro\bin\module.psm1’ was not loaded because no valid module
error> file was found in any module directory.
error> At C:\ProgramData\Syncro\bin\0f7fb8a2-06d8-4999-989e-0fe06131e6ca.ps1:4 char:1
error> + Import-Module $env:SyncroModule -DisableNameChecking
error> + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error> + CategoryInfo : ResourceUnavailable: (C:\ProgramData\Syncro\bin\module.psm1:String) [Import-Module], Fil
error> eNotFoundException
error> + FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand
error>

@Andy Any thoughts on this? This would help me with some issues we run into when scripts can’t run due to execution policy or other problems preventing the script from running in the first place

I don’t think I’ve ever seen this asked before so I am not sure this is something we’d likely do. Is this something you are running into often? If so there is likely a way to smoke test this when you add assets into Syncro so you can correct this out of the gate.

It’s something I mentioned in this thread:

I’ve personally had quite a few issues with scripts silently failing. There are a number of ways a script can fail and still be reported as a success, even if a script is written to handle errors.

  1. Something prevents the script from running, such as execution policy.
  2. A script runs into an unhandled exception/terminating error before it can pass an error code to the parent session.
  3. The module.psm1 file is corrupted or removed by an over-zealous anti-virus solution.

These are all situations I’ve run into, and all situations where Syncro reports a success when the script run is actually failing. All my suggestion is doing is making use of additional information about errors that the system is already providing in the parent PS session, but that we as Syncro customers are not able to access because of how the system currently works.

If you are seeing significant issues here, for those instances I’d go the smoke test route. Consider making a custom field and running a script across all online assets (including when you onboard a new asset as part of the asset policy) and having the script simply write back to the custom field. You could then make a saved asset search for anything that didn’t get written back then it likely has problems that need to be manually triaged.

At any rate we’ll leave up this feature request and see if more folks chime in.

I just had this happen again today. We had an older machine brought online with an old AV solution that keeps sniping the module.psm1 file. All the scripts that run on it are failing, but reporting success.

Except saved asset search cannot search for blank fields can they? So instead you’d be searching for assets where the girls does not match a known value.

The onboarding smoke test still does not cover any other failure that does not result in changing PowerShell’s exit code.

I haven’t found a scenario where I couldn’t somehow test that the work was completed, but finding scripts that fail and are marked as successful is all happenstance, or because there was a critical failure and we trace it back to the script that was supposed to have prevented/responded to a problem ran as a success but actually failed.

They can now, not sure when they snuck it in, but it works. Just leave the field blank and it will match on it.

WHAAAAA :astonished::astonished::astonished:?

clearly I’m not reading release notes well… and or the release notes are missing important features. Thats GREAT

I don’t remember seeing it in release notes. I think Isaac stumbled upon it.