Where the invisible characters come from
None of them are typed on purpose. A chat interface renders the assistant's answer as HTML, and HTML brings non-breaking spaces (U+00A0) around numbers and after some punctuation, zero-width spaces (U+200B) where a long word was allowed to wrap, and soft hyphens (U+00AD) where it was allowed to break. Markdown renderers add variation selectors next to symbols. Word processors and PDFs add typographic spaces (thin, hair, en, em) and curly quotes. Windows adds CRLF line endings and sometimes a byte-order mark (U+FEFF) at the start of a file. Copying text selects all of it, and the pasted result looks identical while comparing, grepping, hashing and compiling differently.
Hidden text and watermarks
Unicode has a block of tag characters (U+E0000–U+E007F), one for each ASCII character, that no font displays. They were meant for language tagging and are now only valid inside a few flag emoji. Because they are invisible but survive copy-and-paste, they are used to hide a second message inside a visible one: a prompt injection in a document an AI assistant will read, a fingerprint identifying who the text was given to, or a way to smuggle instructions past a filter. The same trick is possible with runs of zero-width characters (each one a bit) and with variation selectors (256 of them, so one per byte). The cleaner removes all three and, for tag characters, shows you the decoded hidden text.
What it cannot remove is a statistical watermark, where the pattern lives in the choice of words rather than in any character. There is nothing in the text to strip in that case.
Bidirectional controls
Right-to-left overrides and isolates (U+202E, U+2066–2069) change the display order of what follows. In source code that is the “Trojan Source” attack (CVE-2021-42574): a comment that renders after a condition but is compiled before it. Compilers and GitHub warn about these now; this tool removes them. The Unicode inspector shows each one with its exact position if you need to see rather than strip.
Why some options are off by default
- Smart quotes and dashes are real characters that belong in prose. Turn this on when the destination is code, a config file or a shell, where
“is not"and–is not-. - Look-alike letters (Cyrillic
аfor Latina) are only suspicious in text that is otherwise Latin; the cleaner flags them in that case and leaves a Cyrillic paragraph alone. Replacing them changes letters, so it is a choice, not a default. - Private-use characters are icons from icon fonts most of the time. They show as boxes elsewhere and are harmless.
- Variation selectors are removed only after letters and digits, where they have no job; after emoji and symbols they decide how the glyph renders and are kept.
Doing it on the command line
perl -CSD -pe 's/[\x{200B}-\x{200D}\x{2060}\x{FEFF}\x{00AD}\x{E0000}-\x{E007F}]//g; s/\x{00A0}/ /g' in.txt
sed -i 's/\xC2\xA0/ /g; s/\xE2\x80\x8B//g' file.txt # NBSP and ZWSP as UTF-8 bytes
grep -nP '[^\x00-\x7F]' file.txt # show every non-ASCII line
cat -A file.txt | head # ^M for CR, M-BM- for NBSP Questions
Does this remove AI watermarks?
It removes every character-level mark: zero-width characters, tag characters, variation selectors and unusual spaces, which are the ways text can be fingerprinted invisibly. A watermark encoded in the word choice itself is not made of characters and cannot be removed by any cleaner.
Why does text from ChatGPT or Claude contain hidden characters?
Mostly as a side effect of rendering: the answer is displayed as HTML, and copying HTML carries non-breaking spaces, zero-width spaces at wrap points and typographic punctuation. Some tools and some documents deliberately add tag characters or zero-width runs. The findings list tells you which you have.
Will it break emoji?
No. The zero-width joiner that glues emoji sequences together (👨👩👧) is kept when it sits between two emoji, variation selectors after emoji and in keycaps are kept, and so are the tag characters that make up subdivision flags. Only joiners between ordinary letters are removed.
Is the text uploaded anywhere?
No. The page runs entirely in your browser; nothing is sent to a server. Paste anything.
What is the difference from the Unicode inspector?
The inspector lists every character with its code point, bytes and name. The cleaner groups the problems, explains them and produces the cleaned text. Use the inspector to investigate, the cleaner to fix.