Can I haz full unicode support @forums plz?

[frozenfrogz]

Hello everyone.
I did not really know where I should put this, because I could not find a forum feedback section.

I am running into issues regarding not supported glyphs on a more or less regular basis. This is no big deal in a general sense, but it annoys the hell out of me. As far as I can see it, the site uses the UTF-8 charset, but the editor allows usage of Unicode characters - these are all correctly displayed. This might be a browser-side thing, but IDK. After posting, all characters not part of UTF-8 get replaced by a question mark.

I would love to see either of two options implemented:

A: Full support of unicode characters (preferred)
B: Disallow everything not UTF-8 compliant in the editor.

Is it only me? What is your opinion on that?

Best regards,
Frederik

[ataradov]

Editor is browser-side, it will let you enter whatever your OS allows. The forum then goes and filters stuff that is not ASCII.

[frenky]

In html of site there is charset=UTF-8 tag so that is ok. Probably DB does not support unicode...
Test: ??ž?š
Should be:

EDIT:
Interesting... in preview of the post all characters were ok...

[frozenfrogz]

EDIT:
Interesting... in preview of the post all characters were ok...

And that is the annoying part. Since SOME accents / special chars can be displayed and others can not, it is pretty inconvenient.

acute: é á í ú ó
grave: è à ì ù ò
diaeresis ? ä ï ü ö
Umlaut: ä ü ö
cedilla: ç
scandinavian characters: å æ ø

That is as good as it gets, maybe some Slavic characters, but no Russian etc.

Edit: Noe_el is out as well xD

[cdev]

I could see the accented characters..

[ataradov]

Test of Russian: ? ??.

Nope, not at all. And apparently Russian is a lot of confused smiles

[Zero999]

This has been an issue for as long as the forum has existed and people complain about it, from time to time.

Yes, i do find it odd that some non-ASCII characters are supported and not others. I use the clippings extension for Firefox to keep a list of useful non-ASCII characters, which this site supports. Here they are: µ×÷±¼½¾°

[ataradov]

I think it just supports full extended-ASCII (8-bit): Here is a test of some old-school DOS-style frames: ? ? ?

Nope: frames did not go though. So there is some explicit white list.

<- Here are Cyrillic leters entered as Alt-{numpad 150, 151, 152}. So it does support simple 8-bit set as input.

Nope did not work either. They show up correctly after the posting, but change back to garbage after a page refresh. Strange.

[Monkeh]

I think it just supports full extended-ASCII (8-bit): Here is a test of some old-school DOS-style frames: ? ? ?

Which one is that? 8859-1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16? CP-1252?


‡ ‰ … €

© ª

Looks like perhaps CP-1252. The horror.

I know conversions often don't go well, but.. please. This sucks.

[bitseeker]

It could be a major overhaul to make the forum truly UTF-8. Limitations are OK if it's clear what they are. That's where the preview functionality could do a better job of delivering an accurate preview prior to posting.

At least we can do °C and µV. If only was supported as a character, I think that'd cover the "special" characters I generally use in posts here.

[Bud]

Why is this an issue and why is it needed on an English speaking forum? If people will happily post in their favorite languages it will be a mess.

[ataradov]

Why is this an issue and why is it needed on an English speaking forum? If people will happily post in their favorite languages it will be a mess.

Well, the first time I ran into this problem is when I tried to do a translation of a document for a Soviet-made part someone asked about here.

There are uses for full language support. It is not critical, but in 2017 is important, IMO.

[Monkeh]

Why is this an issue and why is it needed on an English speaking forum? If people will happily post in their favorite languages it will be a mess.

? (up, down, left, right arrows)

¹ ² ³ × ÷ ± (superscript 1, 2, 3, multiply, divide, plus/minus)

?  (Omega)

®  © ™ ((R), (C), TM)

° (degree)

? ¼ ? ½ ? ¾ ? (1/8 1/4 3/8 1/2 5/8 3/4 7/8)

Some gets through. Some doesn't. Remembering what does and doesn't work is annoying.

E: For giggles,a few more by row:
¬!"£$%^&*()_+
|¡?£¼???™±°¿
??E®?¥??ØÞƧЪ???
¦<>©‘’Nº×÷

[Bud]

Honestly, from your list it is not a problem for most of those to use conventional ASCII equivalents and the other ones such as the copyright character may only be needed once in a blue moon.

[Monkeh]

Honestly, from your list it is not a problem for most of those to use conventional ASCII equivalents and the other ones such as the copyright character may only be needed once in a blue moon.

And when I decide to copy and paste something from a document which makes use of these wonderful computers we've developed since the 1960s? I can't tell what's screwed up until it's posted and I can't remember the list of needless limitations.

I really like typing µ and ?. Ah, crap, no ?.

?·m. N·m.

Oh hey, we do get interpunct, that's something.

[Ampera]

Well you can always do

[bluevd]

The problem is much more complicated that it would look at first sight.
Consider the software involved: web server (most likely Apache), script interpreters (PHP), database (MySQL) and dbms (mysqli or whatever php has).
And last but not least, the forum software.
If you ever tried to get the above combo to support everything you know how much of a pain it can be. Sometimes you go through all the pain to realize the forum scripts strip special characters out.
Most likely, you'd be better served by a Pastebin service that supports unicode encoding for posting manual translations and such.
 

[Ian.M]

Actually v2 of SMF forum does support Unicode *IF* it is enabled.  The problem is the message database that is in whatever legacy encoding that was chosen when the forum was originally installed before SMF unicode support.   There is an official conversion procedure but I believe there is a considerable risk of data loss so it would be a lot of work for Gnif to activate it then resolve the resulting problems.

See https://wiki.simplemachines.org/smf/UTF-8_Readme

[rs20]

As far as I can see it, the site uses the UTF-8 charset, but the editor allows usage of Unicode characters.

UTF-8 is an encoding of Unicode, not an alternative. UTF-8 can encode any Unicode character.

The problem is much more complicated that it would look at first sight.
Consider the software involved: web server (most likely Apache), script interpreters (PHP), database (MySQL) and dbms (mysqli or whatever php has).
And last but not least, the forum software.
If you ever tried to get the above combo to support everything you know how much of a pain it can be.

Nope, done it before, it's trivial if you're paying attention. If the mods were to set up a brand new EEVBlog forum from scratch, tomorrow, I bet they'd use UTF-8 throughout without any trouble. But what's really difficult, and the semi-legit reason why this change isn't being made, is that the migration from ASCII to UTF-8 is a huge and risky job -- you have to change all these moving parts in parallel, simultaneously, and also modify the encoding in the database and possibly re-encode everything in the database. It's not the sort of thing you can do casually over a few weekends.

[A Hellene]

Yet, Unicode support is still there for older messages; for example:
https://www.eevblog.com/forum/chat/an-partial-goodbye-could-be/msg42748/#msg42748

How can this be?


-George

[tooki]

Yes, a switch to Unicode is about 15 years overdue. Aside from scientific notation, there are also things like users asking for translations, or phonetic transcription, just to list some examples. It boggles my mind that anyone in this century would configure a forum as ASCII...

[wraper]

Well you can always do

You can, and it's a big PITA. When people ask to translate something about Soviet components or similar, it's a huge time waste. And the worst part is that preview shows everything correctly. So you post just to find a bunch of question marks and smilies. Also, when you replace text with pictures, it's no longer possible to copy and google it, which often would be very useful. For example, to find Russian documentation, or suggest a search term for taobao.

[frozenfrogz]

UTF-8 is an encoding of Unicode, not an alternative. UTF-8 can encode any Unicode character.

Sorry, I got that wrong. So my request would be for full Unicode char set.
Since the forum seems to fully support UTF-8 encoding and can display all Unicode characters (?), the editor might be to blame. Maybe it is just a mismatch of the character encoding in the editor vs. the database.

[A Hellene]

[...]
Maybe it is just a mismatch of the character encoding in the editor vs. the database.

Or, perhaps, some kind of filter between the Editor (that seems to be supporting Unicode during preview) and the Database (that supports Unicode, as can be seen below)?



I do not really know...


-George

[capt bullshot]

Bwargh. Stay away with all that charset confusion.

Anything else but 7 Bit ASCII is to be considered bad!