Windows PowerShell 5.1 REST calls: string bodies go out as Latin-1 and charset-less JSON responses are read as Latin-1

Windows PowerShell 5.1 REST calls and non-ASCII text

Two separate encoding behaviours in Windows PowerShell 5.1 (powershell.exe) corrupt non-ASCII text in HTTP JSON calls. Both are absent in PowerShell 7, which defaults to UTF-8.

Write side: string bodies are encoded as ISO-8859-1

Observed in a bulk migration through a SaaS REST API: records written with Invoke-RestMethod -Method Post -Body $json (and PATCH) lost characters. An en dash became a hyphen, accented letters such as e-acute became the Unicode replacement character U+FFFD when the server read them, and a bullet became a full stop. Reads of the same records decoded correctly, so only data sent by the client was damaged. Six migrated records were affected before the cause was found; they were cosmetic changes (en dash to hyphen) rather than lost data, and records written earlier needed a repair pass.

Reproduction against a local HTTP echo server, string body {"v":"caf<e-acute> <en dash> <bullet>"} and -ContentType 'application/json' (no charset):

| Shell | Bytes the server received | |---|---| | Windows PowerShell 5.1 | 63 61 66 e9 20 2d 20 2e (Latin-1 e-acute, then - and . substituted for the en dash and bullet) | | PowerShell 7.6.9 | 63 61 66 c3 a9 20 e2 80 93 20 e2 80 a2 (UTF-8) |

A server that decodes the body as UTF-8 sees e9 as an invalid sequence and substitutes U+FFFD. The en dash and bullet have no Latin-1 code point, so the best-fit substitution - and . happens on the client, before the data leaves the machine.

Fix

Serialise the body to a string, convert it to UTF-8 bytes, and declare the charset:

$bytes = [Text.Encoding]::UTF8.GetBytes($json)
Invoke-RestMethod -Uri $uri -Method Post -Body $bytes -ContentType 'application/json; charset=utf-8'

In the local test, this form delivered 63 61 66 c3 a9 20 e2 80 93 20 e2 80 a2 from Windows PowerShell 5.1, the same bytes as PowerShell 7. The check used in the migration was a round trip: write a value containing an accented letter and an en dash, read it back, and compare.

One reported variant was not reproduced: the source of this article's first note also listed "UTF-16 bodies by default" as a 5.1 behaviour. The test above showed Latin-1, not UTF-16, for a string body, so the UTF-16 claim is left out.

Read side: a response without a charset is decoded as Latin-1

A JSON response whose Content-Type is application/json with no charset parameter is decoded by Windows PowerShell 5.1 as ISO-8859-1, not UTF-8. In a snapshot cached by a hook script, a UTF-8 em dash (bytes e2 80 94) therefore became the three characters — (shown as â€"). Plain Content-Type: application/json is the normal case for JSON APIs, so this affects most of them.

Local reproduction, response body {"v": "caf<e-acute> <en dash> <bullet>"} encoded as UTF-8 with Content-Type: application/json:

• Windows PowerShell 5.1, Invoke-RestMethod: the string contained code points 99,97,102,195,169,32,226,128,147,... (the UTF-8 bytes as separate Latin-1 characters).
• Windows PowerShell 5.1, Invoke-WebRequest .Content: the same wrong decoding.
• PowerShell 7.6.9: correct code points (233, 8211, 8226).

Decoding the raw stream explicitly gave the right text in 5.1:

$r = Invoke-WebRequest -Uri $uri -UseBasicParsing
[Text.Encoding]::UTF8.GetString($r.RawContentStream.ToArray())

The authors of the original snapshot hook avoided the problem by rewriting it in Python rather than patching the PowerShell. An API that includes charset=utf-8 in its response header is not affected; the migration API in the first note was observed to read correctly, which is consistent with that.

Summary of behaviour

• 5.1 -Body <string> plus a Content-Type without charset: Latin-1 on the wire, lossy for characters outside Latin-1.
• 5.1 response without charset: Latin-1 decoding of UTF-8 text.
• PowerShell 7: UTF-8 on both sides in the same tests.
• Not tested here: -InFile, Invoke-RestMethod with a hashtable body, or other Content-Types such as form data.