Checking to see if anybody else is seeing this. 5 unrelated servers (different clients) are getting random script run failures, each is Server 2019, some beefy specs some not. They all worked fine for months until recently. Now scripts will fail randomly with an error
error> Windows PowerShell terminated with the following error: The type initializer for 'System.Management.Automation.Runspaces.InitialSessionState' threw an exception.
The same script might succeed 5 minutes later. Custom scripts, Syncro provided scripts, CyberDrain scripts.
I’ve updated Windows and installed PowerShell 7, no change. I’m getting HUNDREDS of email notices per day that a script failed and it’s pretty annoying.
The content of the event log includes this:
Creating Scriptblock text (1 of 1):
[redacted contents of Syncro PS1 script here]
ScriptBlock ID: 8f517008-ba7a-4c6d-8e9f-25ba0a02fc2a
Path: C:\ProgramData\Syncro\bin\module.psm1
I think it’s being triggered by the “Import-Module $env:SyncroModule” but I can’t be sure and some scripts that don’t include that line are also failing, just not at the same crazy rate as those that do include that line.
Can you add something like the following near the beginning of the script to see if the Syncro module isn’t being found?
Replace this:
Import-Module $env:SyncroModule
With this:
# MAKE SURE SYNCRO MODULE IS AVIALABLE / ABORT IF NOT
if ($null -eq $env:SyncroModule) {
Write-Output "Syncro module not ready yet... Try again later..."
break
} else {
Import-Module $env:SyncroModule -WarningAction SilentlyContinue
}
Then check the script output on the assets that are failing to see if you get the “Syncro module not ready yet” message in there.
I’ll try this.
If the module isn’t found then it would just produce a normal PS error, would it not? This is not a normal PS error, this is a script being blocked from running. Not the same thing.
I can’t quite tell from the error message you included in your first post, a little more context, or example script would be helpful.
If you’re using Powershell RunSpaces (aka: multi-threading) inside your scripts, it could be a lot of things, including Powershell Execution Policy blocking the loading of any scripts or modules. Syncro overcomes this by loading a runspace with a temporary unrestricted policy, but I don’t think that extends to runspaces spawned inside the session.
If not the above, it’s hard to say without knowing more about the script, or putting more debug code inside the script to see if you can get some script result info that indicates where and why the failures are happening.
If you have specific scripts and their output results when they failed, that might help. Is there more to the output and you only provided the line where it failed, or is the entire output just that one line saying it failed?
The trouble is that it could be ANY script including default ones provided by Syncro, although the most common seem to be CyberDrain’s Dell RAID and their Dell Physical Disks checks. One of those doesn’t call the syncro module at all. Here is a simple one that seems to fail regularly (though still not every time).
Keep in mind that this isn’t a script error per se, otherwise I would expect it to fail every time. This is Windows blocking the execution of the script as a security risk. Powershell Event ID 4104
Was there any script output on the Syncro asset for this one, or you only noticed it getting blocked by looking at the event logs?
Also, I was not familiar with Script Block Logging before today, and have not done a great job at keeping up with the latest Powershell Core developments, which it looks like you might be running, so I might not be very helpful from this point forward. All my Windows machines are running PS v5.x or around there, and I’ve not yet had time to dive into newer versions of Powershell.
If that’s the case then I suspect Syncro is having some kind of problem launching the runspace to run the code, or something in your MS Group Policy is preventing it from being launched.
The problem being intermittent on the same scripts is going to require some digging.
I put the following as the first line in the above script and there was no output. This tells me that Windows blocks the script before any part of it executes. i welcome any alternative interpretations.
I tried one of the scripts that fails regularly (the one pasted above) as a PS1 script on the desktop. Never could get it to fail manually before getting bored from running it over and over.
Here is my updated understanding of the situation now that I’ve been looking at it for a while. The error in the top post is from Powershell terminating BEFORE the script even gets a chance to run the first line. There is always (?) a corresponding event log Powershell/Operational warning 4104, referring to C:\ProgramData\Syncro\bin\module.psm1 and noting the cause being “Execute a Remote Command”. It feels like Windows is totally blocking the script from executing, but it only happens maybe 50% of the time, and only on these 4 Server 2019 machines (and one of them only has the problem maybe 3 times a day instead). This only seems to affect 5 scripts, not all of them. The 5 troublemakers are the one pasted above, a custom script that just checks event log I’ll include below, CyberDrain.com - Monitor Print Queues, CyberDrain.com - Monitor Dell OpenManage Physical Disks, and CyberDrain.com - Monitor Dell OpenManage RAID. I don’t see any commonality between them, but I may not be smart enough to make that claim.
# MAKE SURE SYNCRO MODULE IS AVIALABLE / ABORT IF NOT
if ($null -eq $env:SyncroModule) {
Write-Output "Syncro module not ready yet... Try again later..."
break
} else {
Import-Module $env:SyncroModule -WarningAction SilentlyContinue
}
$subject = "Dell OMSA error or warning detected on " + $env:computername
$Date = (Get-Date).AddDays(-1)
$results = Get-WinEvent -FilterHashTable @{LogName='System' Level=1,2,3;StartTime=$Date;ProviderName='Server Administrator';} -ErrorAction SilentlyContinue
$messages = $subject + $results
if (($results -Like '*A Patrol Read operation started*') -Or ($results -Like '*Patrol Read has started*') -Or ($results -Like '*Patrol Read operation completed*') -Or ($results -Like '*Patrol Read has stopped*')) {Exit 0}
if ($results) {
Rmm-Alert -Category 'dell_omsa_error_warning' -Body $messages
Log-Activity -Subdomain "myorg" -Message "Dell OMSA Warning or Error" -EventName "Dell OMSA Warning or Error"
#Send-Email -To "me@myemail.com" -Subject "Dell OMSA Warning or Error" -Body $messages
}
This is a little bit out of my realm since I’m only running PS 5.1 and Server 2016, so I’m not sure if there are any differences to be expected with Server 2019 or PS 7.x.
I would try to see if you can get back to PS 5.1 and / or reduce your level of script block logging (at least temporarily) to see if that is affecting it somehow.
Here are some articles that talk about script block logging:
But I haven’t had time to fully digest the content to understand what all happens when script block logging is enabled.
The only thing I could find on that error was info from 2016 and a known PS bug that should have been patched unless MS has regressed, which isn’t all that hard to believe.
I was just about to post about this exact error, I started noticing it a couple weeks ago, opened a ticket about it last week, Still waiting for first response from support. I am seeing it on Dozens of servers across a number of clients, All 2019 but thats really the only thing that is consistent. Some servers fail 99% of script runs, others run successfully most of the time.
I can confirm this error is being generated before powershell can actually run my scripts. It happens on all kinds of scripts and not on every run of a given script. Seems to be a problem with Syncro’s script runner and Server 2019, but beyond that I am not sure what else to look at.
it’s not one script and its not every time any of the scripts that are failing runs. Let me grab some screenshots
here is the error returned by every one of these failures:
Like I said I have a bunch of example servers, unfortunately there is no good way to search for all 2019 servers that have not had a script failure with a specific string in the return so I can’t say for sure if it is all, most or just a few 2019 servers.
If there is something else that seems like it would be helpful let me know, but the content of the script never gets to fire.
Curious if there’s some weird security issue. I’ve learned with Syncro and PS that PS uses .net settings. For example, earlier, we were getting script run failures, and had to make a change to .net to resolve them.
Set-ItemProperty -Path ‘HKLM:\SOFTWARE\Wow6432Node\Microsoft.NetFramework\v4.0.30319’ -Name ‘SchUseStrongCrypto’ -Value ‘1’ -Type DWord
What I’m curious about is if either of these options are set to true inside one of these files. NetFx40_LegacySecurityPolicy or LegacyCasPolicy enabled inside of C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config or C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config. I checked and I do not have these on my 2019 boxes, so the option does not exist by default. I checked through all our 2019 systems and none of them are having script failures.
