Every time when I think I finally “got” Unicode, I get kicked in the back by this rabbit hole. 😆 However, IMHO it is important to recognise that when moving data and files between operating systems and programs that you’re better off knowing some of the pitfalls. So I’m sharing something I experienced when I transferred a file to my FreeBSD Play-Around notebook. So let’s assume a little story…
It’s late afternoon and you and some friends sit together playing around with BSD. A friend using another operating system collects coffee orders in a little text file to not forget anyone when going to the barista on the other side of the street. He sends the file to you, so at the next meeting you already know the preferences of your friends. You take a look at who wants a caffè: armin@freebsd:/tmp $ cat orders2.txtMauro: cappuccinoArmin: caffè doppioAnna: caffè shakeratoStefano: caffèFranz: latte macchiatoFrancesca: cappuccinoCarla: latte macchiato So you do a quick grep just to be very surprised! armin@freebsd:/tmp $ grep -i caffè orders2.txtarmin@freebsd:/tmp $ Wait, WAT? Why is there no output? We have more than one line with caffè in the file? Well, you just met one of the many aspects of Unicode. This time it’s called “normalization”. 😎
Many characters can be represented by more than one form. Take the innocent “à” from the example above. There is an accented character in the Unicode characters called LATIN SMALL LETTER A WITH GRAVE. But you could also just use a regular LATIN SMALL LETTER A and combine it with the character COMBINING GRAVE ACCENT from the Unicode characters. Both result in the same character and “look” identical, but aren’t.
Let’s see a line with the word “caffè” as hex dump using the first approach (LATIN SMALL LETTER A WITH GRAVE): \u0063\u0061\u0066\u0066\u00E8\u000Ac a f f è (LF) Now let’s do the same for the same line using the second approach: \u0063\u0061\u0066\u0066\u0065\u0300\u000Ac a f f è (LF) And there you have it, the latter is a byte longer and the two lines do not match up even if both lines are encoded as UTF-8 and the character looks the same!
So obviously just using UTF-8 is not enough and you might encounter files using the second approach. Just to make matter more complicated there are actually four forms of Unicode normalization out there. 😆
NFD: canonical decomposition
NFC: canonical decomposition followed by canonical composition
NFKD: compatible decomposition
NFKC: compatible decomposition followed by canonical composition.
For the sake of brevity of this post and your nerves we’ll just deal with the first two and I refer you to this Wikipedia article for the rest.
Normal form C (NFC) is the most widely used normal form and is also defined by the W3C for HTML, XML, and JavaScript. Technically speaking, encoding in Latin1 (or Windows Codepage 1252), for example, is in normal form C, since an “à” or the umlaut “Ö” is a single character and is not composed of combining characters. Windows and the .Net framework also store Unicode strings in Normal Form C. This does not mean that NFD can be ignored. For example, the Mac OSX file system works with a variant of NFD data, as the Unicode standard was only finalized when OSX was designed. When two applications share Unicode data, but normalize them differently, errors and data loss can result.
So how do we get from one form to another in one of the BSD operating systems (also in Linux)? Well, the Unicode Consortium provides a toolset called ICU — International Components for Unicode. The Documentation URL is unicode-org.github.io/icu/ and you can install that in FreeBSD using the command pkg install icu After completion of the installation you have a new command line tool called uconv (not to be mismatched with iconv which serves a similar purpose). Using uconv you can transcode the normal forms into each other as well do a lot of other encoding stuff (this tool is a rabbit hole in itself 😎).
Similar to iconv you can specify a “from” and a “to” encoding for input. But you can also specify so-called “transliterations” that will be applied to the input. In its simplest form such a transliteration is something in the form SOURCE-TARGET that specifies the operation. The "any" stands for any input character. This is the way I got the hexdump from above by using the transliteration 'any-hex': armin@freebsd:/tmp$ echo caffè | uconv -x 'any-hex'\u0063\u0061\u0066\u0066\u00E8\u000A Instead of hex codes you can also output the Unicode code point names to see the difference between the two forms: armin@freebsd:/tmp$ echo Caffè | uconv -f utf-8 -t utf-8 -x 'any-nfd' | uconv -f utf-8 -x 'any-name' \N{LATIN CAPITAL LETTER C}\N{LATIN SMALL LETTER A}\N{LATIN SMALL LETTER F}\N{LATIN SMALL LETTER F}\N{LATIN SMALL LETTER E}\N{COMBINING GRAVE ACCENT}\N{<control-000A>} Now let’s try this for the NFC form: armin@freebsd:/tmp$ echo Caffè | uconv -f utf-8 -t utf-8 -x 'any-nfc' | uconv -f utf-8 -x 'any-name'\N{LATIN CAPITAL LETTER C}\N{LATIN SMALL LETTER A}\N{LATIN SMALL LETTER F}\N{LATIN SMALL LETTER F}\N{LATIN SMALL LETTER E WITH GRAVE}\N{<control-000A>} You can also convert from one normal form to another by using a transliteration like 'any-nfd' to convert the input to the normal form D (for decomposed, e.g. LATIN SMALL CHARACTER A + COMBINING GRAVE ACCENT) or 'any-nfc' for the normal form C.
If you want to learn about building your own transliterations, there’s a tutorial at unicode-org.github.io/icu/user… that shows the enormous capabilities of uconv.
Using the 'name' transliteration you can easily discern the various Sigmas here (I’m using sed to split the output into multiple lines): armin@freebsd:/tmp $ echo '∑𝛴Σ' | uconv -x 'any-name' | sed -e 's/\\N/\n/g'{N-ARY SUMMATION}{MATHEMATICAL ITALIC CAPITAL SIGMA}{GREEK CAPITAL LETTER SIGMA}{<control-000A>} If you want to get the Unicode character from the name, there are several ways depending on the programming language you prefer. Here is an example using python that shows the German umlaut "Ö": python -c 'import unicodedata; print(unicodedata.lookup(u"LATIN CAPITAL LETTER O WITH DIAERESIS"))' The uconv utility is a very mighty thing and every modern programming language (see the Python example above) also has libraries and modules to support handling Unicode data. The world gets connected, but not in ASCII. 😎
Pyongyang, September 1 (KCNA) — Kim Chon Il, director of the Department of Press and Information of the DPRK Foreign Ministry, on August 31 issued the following press statement “Attempt of the U.S.
Neu im #Blog: „Ist etwas brandgefährlich, bin ich raus.“ Beim Lesen eines Texts fiel das Wort „brandgefährlich“ und ich stieg gedanklich mitten im Satz aus. Das wunderte mich, und darum denke ich kurz darüber nach, wieso das so ist. schreiblehrling.de/ist-etwas-b…
Had a look on Google street view, and, as far as I can tell, this all gone - replaced by generic, suburban-mall-posing-as-classical "architecture". Completely scrubbed of any sign of human activity or place.
By contrast, the names of the businesses in this photo tell you El Paso's priorities in the 70s:
CUT RATE drugs ALLRIGHT auto parts PAYLESS (for what? not specified) LIQUOR (can't be much clearer)
Every inch of it touched and trodden by many real people.
::: spoiler Key Findings 1. Geopolitical Events Trigger Unprecedented DDoS Campaigns Expand: Major political events drove increased DDoS activity, evidenced by attack count spikes that coincide with these occurrences. These events saw hacktivist groups launching up to double the normal number of attacks in short timeframes. 2. Botnet-Driven Attacks Dominate with Increased Sophistication Expand: Botnet-driven attacks are getting longer, more frequent, and are employing multiple attack vectors to avoid mitigation. They are targeting known vulnerabilities in IoT devices, servers, routers, and more. 3. NoName057(16) Maintains Dominance Among Familiar Threat Actors: Well-known hacktivist and attack groups, such as NoName057(16), are launching more attacks across the globe while leveraging several attack vectors. 4. New Threat Actors Emerge with DDoS-as-a-Service Capabilities: Emerging attack groups like DieNet and Keymous+ are leveraging DDoS-for-hire infrastructure to launch DDoS-as-a-service campaigns, lowering the barrier to entry and expanding the threat landscape. 5. Global DDOS Attack Volume High with Regional Variations: With more than 8 million recorded attacks globally in the first half of 2025, DDoS attack volume remains massive. The attacks also show sustained intensity, reaching speeds of 3.12 Tbps and 1.5 Gpps. :::
DDoS attacks are no longer just a nuisance, they’re a weapon of geopolitical influence. In the first half of 2025 alone, more than 8 million attacks were recorded globally, with threat actors leveraging AI, botnets, and DDoS-for-hire services to launch increasingly sophisticated and sustained campaigns.
@mpfpr holds 4th workshop for members of National Constitutional Review Commission (NCRC) 🇸🇸 on ‘Constitutionalism and Federalism’. Generously supported by the #EuropeanUnion
"Babe, what's up?" Me, wondering whether Prince had a nice time on Millennium Eve, or if his expectations had become so high that the reality of any party, no matter how good, could only disappoint: "Nothing, babe."
“A group of white people from immigrant backgrounds have held a series of marches across Australia pleading for their own community to leave Australia at once.
The idea that the protesters should “go back to where they came from” has been met with agreement across vast swathes of the Australian public.”
theshovel.com.au/2023/01/26/mi… “Boat people from as far back as 1788 didn’t bother applying for passports or visas, were set on introducing their own laws, and, in many cases concealed criminal records, according to the damning report.
Chief researcher Marianne Williams said migrants have stubbornly refused to adapt to the local culture. “They haven’t made an effort to fit in, and in a lot of cases they’ve tried to impose their own laws onto those already here”.
Da ich zurzeit noch ein AppleTV+ Abo, schaue ich ab und an mal, was es dort an Filmen gibt. Gestern bin ich über The Gorge gestolpert. Die ersten Minuten haben zwei entspannte Stunden versprochen, in denen ich etwas lesen oder mich um ...
Big Tech does everything in its power to hide and deceive when it comes to its true environmental impact. It's been aggressively greenwashing since the 1950s when in Silicon Valley it hid the chemical tanks underground that were needed to make its chips. From there we got to the Cloud and an industry of lies and deceit.
As demand for artificial intelligence technology boosts construction and proposed construction of data centers around the world, those computers require not just electricity and land, but also a significant amount of water.
Jumped into cleaning up a rusty, paint-spattered herd of chisels acquired in winter. One needs a new handle for sure. 5€ for the lot, thereabouts. A worthy exploration in any case.
Scraped and bathed. Now working on the refinishing. Sharpening will be later in year.
LtR: two unmarked and a Muller, two Alchrom and a J-C Brune, a ¾” (Chinese), two Stanley (black handles), and a Peugeot. All mid-20th +/-, I’d guess.
End of thread. The prerequisite before and after shots. I’m about to oil them, but you know how that goes.
I wanted to get these cleaned up because I’ll soon be making chisel shelves and didn’t want rusty chisels soiling everything.
Several other chisels I have need similar handle work, but I cleaned their blades months ago. I’ll get back to those (and these) when I start grinding/honing edges.
I’ve a few more firmers to share prep, but first back to the Frankenstein saw.
Diritto allo studio e libri scolastici: “Incrementi in linea con l’inflazione” @libri illibraio.it/news/editoria/dir… Ancora una volta, si discure di "caro-scuola". Nel 2025 la spesa media per i libri è di 190 euro l’anno nelle scuole secondarie di primo grado, 279 euro l’anno nei licei, 241 negli istituti tecnici, 167 negli istituti professionali. Per Giorgio Riva, presidente del gruppo Educativo dell'Associazione Italiana
Profitez des vidéos et de la musique que vous aimez, mettez en ligne des contenus originaux, et partagez-les avec vos amis, vos proches et le monde entier.
Non so, ho un po' di mixed feelings qui. Da una parte non si può non sorridere vedendo Pike con il capello che sta per dichiarare l'indipendenza e Patton Oswalt in versione vulcaniana. Dall'altro questo continuo puntare sulla commedia temo nasconda una certa carenza di idee. #starTrek #StrangeNewWorlds
The image features a man with silver-gray hair styled upwards, wearing a detailed, ornate brown vest adorned with various gemstones and metallic accents. The vest has a textured appearance with intricate designs, including red and blue gemstones set in a metallic frame. The man's expression is serious, with furrowed brows and a stern look. He is standing in a futuristic setting, with a blurred background that includes white walls and blue lighting, suggesting a high-tech environment, possibly a spaceship or a futuristic facility. The lighting is bright, highlighting the man's features and the details of his attire.
Provided by @altbot, generated privately and locally using Ovis2-8B
Η ΚΕΕΡΦΑ (Κίνηση Ενωμένοι Ενάντια στον Ρατσισμό και τη Φασιστική Απειλή) καταγγέλλει ρατσιστική εργοδοτική βία σε βάρος μετανάστη εργάτη. Ο μετανάστης Ραφίκ Μοχάμεντ δέχτηκε επίθεση με σιδηρολοστό από άτομο που…
"[...] esistesse una rete organizzazioni contro i diritti, finanziata da ingenti somme di denaro provenienti da Russia e USA e pronta a collaborare con partiti di destra ed estrema destra in ascesa a livello globale".
Nel 2017 Wikileaks rivelò “Agenda Europe”, un manifesto che delineava la strategia di una rete transatlantica ultra cattolica per erodere diritti come aborto, divorzio e libertà LGBTQ+.
The image shows a street scene with two tractors parked in front of a building. The building has a white facade with black roofing and multiple windows, some with flower boxes. The tractors are positioned near a stop sign, with one red and the other cream-colored. The red tractor is on the left, and the cream-colored one is on the right. A gray pickup truck is parked behind the cream-colored tractor. The street is paved, and there is a landscaped area with green grass in front of the tractors. The sky is overcast, suggesting a cloudy day. The storefronts on the ground floor include "Metlys" and "entia."
Provided by @altbot, generated privately and locally using Ovis2-8B
#Spotted on the First Day of Spring in Aotearoa New Zealand:
A Shetland pony stallion is prancing around a daffodil lined paddock while snorting and neighing his amorous intentions to a big chestnut mare. He may be tiny but his adoration is MIGHTY. Spring's in the air and he's in the mood for BIG love.
A small human (3?), wearing bright blue waist-high waders is helping Grandad fish in a pretty pebble-lined river. It's snowing on the mountains in the distance and a curious weka* is inspecting their tackle box, but Small Human cares not. He and Grandad are here to fish and that is serious business.
A man (60s?) wearing old Levis, bare feet and a faded Shihad* T-shirt is whistling softly to a tūī* that's camped out in a nearby blossoming plum tree. Holding a steaming cup of coffee and smiling into the sun as birdsong fills his garden.
A woman (50s?) is talking to a friend as they sit in a cafe sunbeam. She says: "You know, Dad's always been such a big tough old man but you wouldn't BELIEVE how much he loves his little dog. They're inseparable!"
A man (50s?) greets a friend in a quiet street. "Gotta get home before the rain, bro! We've got the whānau* commin' tonight and I've gotta get the lawn mowed! It's gonna be a good one!" His friend asks him if he needs a hand and they both walk off together.
Any typos spotted in this post had a lovely first day of spring. The trees are beginning to blossom. Birds are singing. And a very determined pony spotted on their evening walk left them belly laughing as they watched him attempt to (unsuccessfully) woo a lady love 10 times his size.
---
*Weka are a brown woodhen. Native to Aotearoa. Much loved. Very sneaky. Known for stealing anything not nailed down and legging it. (Writes an author who recently had a banana snaffled by a very sneaky bird while tramping.)
** Shihad is a legendary classic Kiwi rock band. (Incidentally the first band the typos saw with their lovely partner Tony in a university bar in 2000.) Here's some unadulterated millennial magnificence for your enjoyment. youtube.com/watch?v=K9k5UC2w25…
***Tūīs are native New Zealand birds with a very distinctive call. They are blue, green and bronze with a throat tuft of white feathers that looks a little bit like a fancy cravat.
Profitez des vidéos et de la musique que vous aimez, mettez en ligne des contenus originaux, et partagez-les avec vos amis, vos proches et le monde entier.
Venerdì 5 e sabato 6 settembre 2005, dalle 10 alle 18 – e domenica 7, dalle 10 alle 14 a Roma, presso lo Studio Campo Boario Viale del Campo Boario 4/a _ (Metro B – fermata Piramide) ESISTE LA RICERCA quinto incontro (2025): IL GIOCO DELLE COMU…
Venerdì 5 e sabato 6 settembre 2005, dalle 10 alle 18 – e domenica 7, dalle 10 alle 14 a Roma, presso lo Studio Campo Boario Viale del Campo Boario 4/a _ (Metro B – fermata Piramide) ESISTE LA RICERCA quinto incontro (2025): IL GIOCO DELLE COMU…
Profitez des vidéos et de la musique que vous aimez, mettez en ligne des contenus originaux, et partagez-les avec vos amis, vos proches et le monde entier.
"India’s digital ambitions and climate commitments need not be at odds ... The development of Green Energy Corridors have already begun to reshape India’s power landscape. These policies are gradually enabling digital infrastructure, particularly data centers, to transition toward solar, wind, and other renewable energy sources" sikkimexpress.com/news-details…
Here lies the insidious allure of renewables. Consume what you want, once you're using renewables. There's no environmental cost to green energy
I have just finished preparing for the arrow-building process, i.e. deciding which tools are necessary for making wooden and carbon arrows, and which steps are involved. This is only the first step: preparation. I will start building once I have received all the materials.
Global sumud flotilla: partite le prime barche. altre se ne aggiungeranno giovedÌ da catania @anarchia Domenica 31 agosto sono salpate da Genova e Barcellona le prime imbarcazioni della Global Sumud Flotilla, una flotta civile internazionale decisa a rompere l’assedio imposto da Israele alla Striscia di Gaza. Da Tel Aviv, il ministro dell’ultradestra rivoluzioneanarchica.it/global…
)
Quincy
in reply to Irritationshintergrund🍑 • • •