Accent problem when running python script straight from C#

Question

I’m using this to run the code:

processo.StartInfo = new System.Diagnostics.ProcessStartInfo
{
    FileName = @"\Python27\python.exe",
    Arguments = arquivoResposta,
    UseShellExecute = false,
    CreateNoWindow = true,
    WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden,
    RedirectStandardOutput = true,
    RedirectStandardError = true,
    RedirectStandardInput = true
};

Here’s an example of the correct output:

Média mensal de atendimentos: 13.17
----
Mês 5: 17
Mês 7: 14
Mês 8: 14
Mês 11: 23
Mês 12: 30

What is coming out:

Média mensal de atendimentos: 13.17

----

Mês 5: 17

Mês 7: 14

Mês 8: 14

Mês 11: 23

Mês 12: 30

Does anyone have a solution? I also tried using output.normalize(), but still nothing.

Answer 1

What happens is that when you run a Python script straight through the command, it can detect the terminal encoding (which is "cp-852") and encode the output accordingly.

When you run from another process, Python cannot guess the encoding of the terminal, and uses utf-8 by default. This is if it is Python3, or if you used Unicode strings correctly, or close to correct in Python2 - otherwise it may be the case that you are using utf-8 hardcoded.

When you try to display accented text, encoded in utf-8 in a Latin-1 output, the accentuation is displayed wrong in the way you posted (each accented character unfolds in two, the first being an "Is" )

First you have to understand what is happening when it comes to encoding accents. Do not delay question,r copy and paste answers, or try to keep changing parameters to "Encode" and "Decode" at random. This article has a good introduction.

If your Python2 program is using byte-strings, then it is hard-coded to utf-8 even - and if you run it straight from the command you will see the wrong accent, but otherwise.

So, second thing - try using Python 3.6.0 or 3.5 instead of Python 2.7- there is no point using such an old version of Python in a new project, and the biggest change from Python2 to Python3 is precisely the automatic handling of accent encoding in some of the cases.

If the problem persists, it is because, as I said in the first paragraph, there is no way that Python can actually guess the encoding of its output from the data that C# sends to it in the redirect. In this case, you must force the encoding to latin-1. It is very unlikely that this is the case, because it is very difficult to force Python’s "print" to use another encoding for stdout other than the one that is there. What I believe to be your case, as in Python2, is that you have used byte-strings (that is, without the prefix u"..." in Python2) and Tar with your hardcoded source file in utf-8. By switching to Python3, the thing is solved.

Otherwise, another way to solve is to save your.py file with the "latin1" encoding directly. In this case, the change will only be made in your programming text editor, and the first or second line indicating the file encoding must be changed from "# coding:utf-8" to "# coding: latin1". Your program will work without changes, but you will lose many of the word processing tools correctly - so the tip to switch to Python 3.