Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • U uchardet
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 14
    • Issues 14
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • uchardet
  • uchardet
  • Issues
  • #7
Closed
Open
Issue created Oct 22, 2018 by Zhiming Wang@zmwangx

UTF-16BE/UTF-16LE without BOM not supported (?)

I took some random UTF-8 encoded paragraph of Chinese text from https://zh.wikipedia.org and converted it to UTF-16, UTF-16BE, UTF-16LE with iconv (GNU libiconv 1.11 on macOS 10.14). The UTF-16BE and UTF-16LE version have no BOM, and in particular, the 2-byte BOM is the only difference between the UTF-16 and the UTF-16BE version. Rather surprisingly, uchardet failed on both the UTF-16BE and UTF-16LE versions:

$ ./src/tools/uchardet zh.utf-8.txt zh.utf-16.txt zh.utf-16be.txt zh.utf-16le.txt
zh.utf-8.txt: UTF-8
zh.utf-16.txt: UTF-16
zh.utf-16be.txt: unknown
zh.utf-16le.txt: WINDOWS-1252

I have attached the text files.

Is there any chance this could be improved?

zh.utf-8.txt zh.utf-16.txt zh.utf-16be.txt zh.utf-16le.txt

Assignee
Assign to
Time tracking