Windows PowerShell 5.1 and BOM-less UTF-8: Get-Content, Set-Content and .ps1 files are read as the ANSI code page

Windows PowerShell 5.1 and BOM-less UTF-8 files

Windows PowerShell 5.1 (powershell.exe) treats a file with no byte order mark as being in the system ANSI code page (Windows-1252 on a Western-European or English system), not UTF-8. PowerShell 7 treats it as UTF-8. The same file therefore behaves differently in the two shells. Everything below was observed on a Windows-1252 machine; other ANSI code pages were not tested.

Get-Content and Set-Content

Test file: the text a <em dash> b caf<e-acute> saved as UTF-8 without a BOM (bytes e2 80 94 for the dash, c3 a9 for the e-acute).

• Get-Content file in 5.1 produced the code points 226, 8364, 8221 for the dash and 195, 169 for the e-acute, which print as the familiar mojibake — and é. PowerShell 7 reads the same file correctly.
• Get-Content file | Set-Content out (both defaults) wrote back exactly the original bytes in the test. The mojibake characters were re-encoded to Windows-1252, which reverses the misreading, so a pure copy was lossless for these characters.
• Get-Content file | Set-Content out -Encoding UTF8 wrote ef bb bf (a BOM) followed by double-encoded text: the dash became c3 a2 e2 82 ac e2 80 9d. This is the damaged result: every em dash becomes the three visible characters â€".
• Any in-memory edit between read and write (a -replace, a template fill) makes the misreading visible to the program and, if the file is written with -Encoding UTF8 or sent over the network, permanent. A character outside Windows-1252 that arrives in the string (for example an arrow, U+2192) is written by default Set-Content as 3f (?), so it is lost.
• A file that has a BOM is read correctly by Get-Content. Writing the correctly read text back with the default Set-Content then produces a Windows-1252 file, not UTF-8 (the dash became the single byte 97, the e-acute e9), which breaks any UTF-8 consumer.

In an infrastructure log, an em dash was mangled this way after a repeated edit with Get-Content/Set-Content; the recorded fix was to use .NET directly with an explicit encoding:

$utf8 = New-Object Text.UTF8Encoding($false)   # no BOM
$text = [IO.File]::ReadAllText($path, $utf8)
# ... edit $text ...
[IO.File]::WriteAllText($path, $text, $utf8)

In the re-test, ReadAllText($path, $utf8) returned the correct em dash (code point 8212). Notably, [IO.File]::ReadAllText($path) with no encoding argument also returned the right text, because .NET defaults to UTF-8 detection rather than the ANSI code page.

One additional report from a project note (NUL bytes appearing in a .env file after a Get-Content/Set-Content round trip, with every API call then returning 401) could not be traced to a source and was not reproduced: a UTF-16 file rewritten through Get-Content/Set-Content produced a clean ANSI file in the test, with no NUL bytes. The report is mentioned only so readers do not treat it as established.

.ps1 script files without a BOM

Windows PowerShell 5.1 reads a BOM-less UTF-8 script as Windows-1252 as well. A non-ASCII character that appears inside a code string is decoded as several characters, and one of those can be a curly quote that the parser accepts as a string delimiter.

Reproduction: a script saved as UTF-8 without BOM, containing $x = "a <en dash> b". Windows PowerShell 5.1 stops with:

The string is missing the terminator: ".

reported at a later line than the one that holds the dash (here the closing line of the script, with a second error pointing at the dash line). The en dash is e2 80 93; the byte 93 in Windows-1252 is a left double quote. An em dash (e2 80 94, where 94 is a right double quote) failed the same way. The same script ran normally under PowerShell 7.

A non-ASCII character in a comment did not break parsing in the test, since the decoded characters stay inside the comment.

Options that avoid the failure:

• Keep script code ASCII (use [char]0x2014 or a \u escape in another language instead of the literal character).
• Save the file with a UTF-8 BOM, which Windows PowerShell 5.1 honours.
• Run the script with PowerShell 7.

Not covered

• Behaviour with other ANSI code pages, with UTF-16 scripts, or on the Windows beta "Use Unicode UTF-8 for worldwide language support" setting (which changes the system ANSI code page to UTF-8) was not tested.
• Piping text to native commands also adds a BOM in some setups; that is a separate issue.