I had a string of data containing French letters, that needed to be saved as XML for syndication. A particular protocol e. Microsoft conventions for. When you need to conform to such a protocol, use a BOM. Where a text data stream is known to be plain text, but of unknown encoding, BOM can be used as a signature. If there is no BOM, the encoding could be anything. Where a text data stream is known to be plain Unicode text but not which endian , then BOM can be used as a signature.
If there is no BOM, the text should be interpreted as big-endian. Where the precise type of the data stream is known e. Unicode big-endian or Unicode little-endian , the BOM should not be used.
Invariably the layout would get destroyed when saving. It took my some time to fiddle my way around this. Never again. Some authorities recommend against using the byte order mark in POSIX Unix-like scripts,[15] for this reason and for wider interoperability and philosophical concerns.
The byte order mark BOM is a Unicode character used to signal the endianness byte order of a text file or stream. BOM use is optional, and, if used, should appear at the start of the text stream.
Beyond its specific use as a byte-order indicator, the BOM character may also indicate which of the several Unicode representations the text is encoded in. My real problem with the absence of BOM is the following.
Suppose we've got a file which contains:. So another user of this file opens it and appends some native characters, for example:. This is not UTF-8 and this causes other problems later on in the development chain. Here is my experience with Visual Studio, Sourcetree and Bitbucket pull requests, which has been giving me some problems:. So it turns out BOM with a signature will include a red dot character on each file when reviewing a pull request it can be quite annoying.
If you hover on it, it will show a character like "ufeff", but it turns out Sourcetree does not show these types of bytemarks, so it will most likely end up in your pull requests, which should be ok because that's how Visual Studio encodes new files now, so maybe Bitbucket should ignore this or make it show in another way, more info here:. Red dot marker BitBucket diff view. How are we doing? Please help us improve Stack Overflow.
Take our short survey. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Ask Question. Asked 11 years, 11 months ago. Active 1 year, 9 months ago. Viewed k times. Otherwise assume that it is CP or some other 8 bit encoding.
Scanning large files for UTF-8 content takes time. A BOM makes this process much faster. In practice you often need to do both. The culprit nowadays is that still a lot of text content isn't Unicode, and I still bump into tools that say they do Unicode for instance UTF-8 but emit their content a different codepage.
Tronic I don't really think that "better" fits in this case. It depends on the environment. This is just one of those Microsoft naming lies, like calling an encoding "Unicode" when there is no such thing. To detect a UTF-8 byte sequence it may be useful to note that the first byte of a multi-byte sequence "codepoint" the bytes that are NOT "plain" ASCII ones has the MS bit set and all one to three more successively less significant bits followed by a reset bit.
Show 11 more comments. Active Oldest Votes. Peter Mortensen Martin Cote Martin Cote Regardless of it not being recommended by the standard, it's allowed, and I greatly prefer having something to act as a UTF-8 signature rather the alternatives of assuming or guessing.
But most of us living in the real world can't change the file system of the OS s our programs get run on -- so using the Unicode standard's platform-independent BOM signature seems like the best and most practical alternative IMHO. What's unfortunate is that the ones responsible for the immense amount of pain cause by the UTF-8 BOM are largely oblivious to it. Show 24 more comments. Alcott : You understood correctly.
You need external information to choose how to interpret it. If you don't know, then you must try to find out. The BOM could be a clue. The absence of invalid character when decoded as UTF-8 could be another Also you can never know.
Conclusion: It is not latin-1 with a certainty well above the certainty without it. But if your system relies on guessing , that's where uncertainties come in. Some malicious user submits text starting with these 3 letters on purpose, and your system suddenly assumes it's looking at UTF-8 with a BOM, treats the text as UTF-8 where it should use Latin-1, and some Unicode injection takes place.
Just a hypothetical example, but certainly possible. You can't judge a text encoding by its content, period. In other words: either standardize your content and say, "We're always using this encoding. Write it that way. This single 4 code unit corresponds to the Unicode scalar value, which is the abstract number associated with a Unicode character.
For more information, see Section 3. A: This depends. However, the downside of UTF is that it forces you to use bits for each character, when only 21 bits are ever needed.
The number of significant bits needed for the average character in common texts is much lower, making the ratio effectively that much worse. In many situations that does not matter, and the convenience of having a fixed number of code units per character can be the deciding factor. These features were enough to swing industry to the side of using Unicode UTF While a UTF representation does make the programming model somewhat simpler, the increased average storage size has real drawbacks, making a complete transition to UTF less compelling.
With UTF APIs the low level indexing is at the storage or code unit level, with higher-level mechanisms for graphemes or words specifying their boundaries in terms of the code units.
This provides efficiency at the low levels, and the required functionality at the high levels. If its ever necessary to locate the n th character, indexing by character can be implemented as a high level operation. However, while converting from such a UTF code unit index to a character index or vice versa is fairly straightforward, it does involve a scan through the bit units up to the index point. While there are some interesting optimizations that can be performed, it will always be slower on average.
Therefore locating other boundaries, such as grapheme, word, line or sentence boundaries proceeds directly from the code unit index, not indirectly via an intermediate character code index. A: Almost all international functions upper-, lower-, titlecasing, case folding, drawing, measuring, collation, transliteration, grapheme-, word-, linebreaks, etc.
Single code-point APIs almost always produce the wrong results except for very simple languages, either because you need more context to get the right answer, or because you need to generate a sequence of characters to return the right answer, or both.
Trying to collate by handling single code-points at a time, would get the wrong answer. The same will happen for drawing or measuring text a single code-point at a time; because scripts like Arabic are contextual, the width of x plus the width of y is not equal to the width of xy.
In particular, the title casing operation requires strings as input, not single code-points at a time. In other words, most API parameters and fields of composite data types should not be defined as a character, but as a string.
And if they are strings, it does not matter what the internal representation of the string is. Both UTF and UTF-8 are designed to make working with substrings easy, by the fact that the sequence of code units for a given code point is unique. Q: Are there exceptions to the rule of exclusively using string parameters in APIs? A: The main exception are very low-level operations such as getting character properties e.
As one 4-byte sequence or as two 4-byte sequences? A: The definition of UTF requires that supplementary characters those using surrogate pairs in UTF be encoded with a single 4-byte sequence. A: If an unpaired surrogate is encountered when converting ill-formed UTF data, any conformant converter must treat this as an error. By representing such an unpaired surrogate on its own, the resulting UTF data stream would become ill-formed.
While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in valid data stream. Under some higher level protocols, use of a BOM may be mandatory or prohibited in the Unicode data stream defined in that protocol. A: A BOM is useful at the beginning of files that are typed as text, but for which it is not known whether they are in big or little endian format—it can also serve as a hint indicating that the file is in Unicode, as opposed to in a legacy encoding and furthermore, it act as a signature for the specific encoding form used.
A: Data types longer than a byte can be stored in computer memory with the most significant byte MSB first or last. The former is called big-endian, the latter little-endian. When data is exchanged, bytes that appear in the "correct" order on the sending system may appear to be out of order on the receiving system. In that situation, a BOM would look like 0xFFFE which is a noncharacter , allowing the receiving system to apply byte reversal before processing the data.
UTF-8 is byte oriented and therefore does not have that issue. In that form, the BOM serves to indicate both that it is a Unicode file, and which of the formats it is in. If yes, then can I still assume the remaining UTF-8 bytes are in big-endian order? However, it makes no difference as to the endianness of the byte stream. UTF-8 always has the same byte order. Q: I am using a protocol that has BOM at the start of text. A: Where the data has an associated type, such as a field in a database, a BOM is unnecessary.
Do not tag every string in a database or set of fields with a BOM, since it wastes space and complicates string concatenation. Moreover, it also means two data fields may have precisely the same content, but not be binary-equal where one is prefaced by a BOM. I copied the resulting file to the server, and to my amazement, it worked. I'm pleased that I've at least found a solution to the problem, but it would be much easier to use an editor that provides an option to save UTF -8 without the BOM.
Any suggestions? Posted by Pam Berman on Wednesday, 25th May at It's a good Unicode editor with all the options you could possibly want for character encoding. It's also free. Posted by Hans on Wednesday, 25th May at Thank you Pam and Hans. Posted by Gez on Wednesday, 25th May at I totally agree, Gez. BabelPad is much more user-friendly. Posted by Pam on Wednesday, 25th May at Posted by holly on Friday, 27th May at Hey Gez, an old student of the ND here, probably won't remember me.
Posted by Matthew on Friday, 27th May at Learn more. Asked 9 years, 2 months ago. Active 9 months ago. Viewed 65k times. Add a comment. Active Oldest Votes. You can open with codecs : import json import codecs json.
Pavel Anossov Pavel Anossov I strongly recommend using io. The io module is more robust and faster. MartijnPieters: Thanks for that comment, good to know. I found this discussion of the differences that might be useful: groups. You don't even need to import codecs. John R Perry 3, 2 2 gold badges 34 34 silver badges 50 50 bronze badges.
0コメント