When sharing files containing special characters like ()?: through Samba, Windows clients often display mangled filenames (e.g., my_file:_(important).txt becomes M43J1E~0.TXT). This occurs because Samba's default behavior translates "illegal" Windows characters into DOS-compatible 8.3 names.

This isn't an encoding issue (UTF-8 works fine for non-special characters) but rather Samba's mangled names feature attempting to maintain compatibility with legacy Windows filesystems. The problematic characters include:

\\ / : * ? " < > |

Add these parameters to your Samba config (smb.conf):

[global]
    mangled names = no
    unix extensions = no
    vfs objects = catia
    catia:mappings = 0x22:0xa8,0x2a:0xa6,0x3a:0xa3,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6

The catia VFS module helps translate these characters to Windows-acceptable alternatives.

After modifying smb.conf, verify with:

testparm -s

Then restart Samba:

sudo systemctl restart smbd

For Windows clients that must access existing shares with these characters, modify the registry (use at your own risk):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters]
"AllowInsecureGuestAuth"=dword:00000001
"RequireSecureNegotiate"=dword:00000000

Here's how a filename transforms under different configurations:

Original: project_final_(v2.0).docx
Default Samba: PROJEC~1.DOC
With our solution: project_final_(v2.0).docx

When working with Samba shares, you might encounter an annoying behavior where filenames containing certain special characters get converted to DOS-style 8.3 shortnames. This particularly affects characters like:

:( ) ? " < > | *

For example:

Original: my_file:_(important).txt
Converted: M43J1E~0.TXT

This isn't actually an encoding issue - Samba handles UTF-8 filenames just fine. The conversion happens because:

1. Samba defaults to creating DOS-compatible shortnames for Windows clients
2. Some characters are forbidden in Windows filenames
3. The conversion algorithm prioritizes compatibility over readability

Add these parameters to your smb.conf in the [global] section:

[global]
# Disable shortname creation
mangled names = no

# Allow all characters except those forbidden by Windows
valid chars = :?()

If you can't modify the Samba server configuration, try these client-side approaches:

# Mount with UNIX extensions (Linux clients)
mount -t cifs //server/share /mnt -o noserverino,unix

After making changes, verify with:

testparm -s
smbclient //localhost/share -U user

For read-only shares, you could create symlinks to problematic filenames:

ln -s "my_file:_(important).txt" clean_name.txt

• Windows clients may still have issues with certain characters
• The 'valid chars' parameter affects all shares
• Legacy Samba versions might need additional tuning