Convert ZSH history file (.zsh_history file) between original weird encoding and proper UTF-8.

zshhist.py

#!/usr/bin/env python3

"""
ZSH history convert script.

When you mistakenly type your password or quite bad command into ZSH,
you may want to remove the entry from the history.
The problem is, the .zsh_history is encoded in a weird format!
If you want to find a command with non-ASCII character, it'll be problematic.

Here is the small script to convert between .zsh_history and valid UTF-8 file.
You can follow the steps shown below to manipulate your .zsh_history.

$ python3 zshhist.py export > /tmp/exported
$ vim /tmp/exported
$ python3 zshhist.py import /tmp/exported >! ~/.zsh_history

Referring: http://www.zsh.org/mla/users/2011/msg00155.html

MIT License

Copyright (c) 2012-2023 Katsuaki Ikegami

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""

import os
import sys
from argparse import ArgumentParser, FileType as OriginalFileType


DEFAULT_ZSH_HIST_FILE = "/path/no/exists" # os.path.join(os.environ["HOME"], ".zsh_history")


class FileType(OriginalFileType):

    """Bug-fix to original argparse FileType factory.

    This Factory returns buffer IO wrapper when binary mode is set.
    """

    def __init__(self, *args, **kwargs):
        super(FileType, self).__init__(*args, **kwargs)

    def __call__(self, string):
        if string == '-' and 'b' in self._mode:
            if 'r' in self._mode: return sys.stdin.buffer
            if 'w' in self._mode: return sys.stdout.buffer
        return super(FileType, self).__call__(string)

def mapNotNone(function, iterable):
    return filter(lambda x: x is not None, map(function, iterable))


def ismeta(ch):
    return ((ch > 0x83 and ch < 0x9e)
            or ch == 0xa0 or ch == 0x83 or ch == 0)


def readhist(bs):
    change = False
    result = bytearray()
    for c in bs:
        if c != 0x83:
            if change:
                d = c ^ 32;
            else:
                d = c
            result.append(d)
            change = False
        else:
            change = True
    return bytes(result)


def writehist(bs):
    result = bytearray()
    for c in bs:
        if ismeta(c):
            d = 0x83
            result.append(d)
            d = c ^ 32
            result.append(d)
        else:
            result.append(c)
    return bytes(result)


class InvalidFormatError(Exception):
    def __init__(self, index, byte):
        self.index = index
        self.byte  = byte

    def __str__(self):
        return "InvalidFormatError(line={0.index}, byte={0.byte})".format(self)


def byte2u(index, bs, strict):
    try:
        return bs.decode("utf-8")
    except UnicodeDecodeError:
        if strict:
            raise InvalidFormatError(index, bs)
        else:
            return None


def handle_export(args):
    """Export ZSH history into valid UTF-8, emit to stdout.

    Parameters:
        args: object with following properties.
              * source (Reader): input ZSH history file.
              * strict (bool): True to make behavior strict.
    """

    def f(arg):
        i, x = arg
        return byte2u(i, readhist(x), args.strict)

    try:
        hists = mapNotNone(f, enumerate(args.source.readlines()))
    except InvalidFormatError as e:
        print("Invalid characters @ line {0}".format(e.index+1),
              file=stderr)
        exit(2)
    sys.stdout.write("".join(hists))


def handle_import(args):
    sys.stdout.buffer.write(writehist(args.source.read()))


def handle_unknown(parser, args):
    parser.print_help(sys.stderr)
    sys.exit(2)


def main():
    parser = ArgumentParser(description="zsh history UTF-8 converter")
    parser.set_defaults(func=lambda args: handle_unknown(parser, args))

    subparsers = parser.add_subparsers()

    parser_read = subparsers.add_parser("export",
                                        help="export zsh history file to UTF-8 text, emit to standard output.")
    parser_read.add_argument('--strict', action='store_true', default=False,
                             help="strict mode (default: no)")
    parser_read.add_argument('source', nargs='?', type=FileType("rb"),
                             default=DEFAULT_ZSH_HIST_FILE,
                             help="target file (default: $HOME/.zsh_history)")
    parser_read.set_defaults(func=handle_export)

    parser_write = subparsers.add_parser("import",
                                         help="import UTF-8 text into zsh history format, emit to standard output.")
    parser_write.add_argument('source', nargs='?', type=FileType("rb", 0),
                              default=sys.stdin.buffer,
                              help="target file (default: stdin)")
    parser_write.set_defaults(func=handle_import)

    args = parser.parse_args()
    args.func(args)

    return 0

if __name__ == "__main__":
    main()

(日本人の方と確信したので日本語で)
素晴らしいです。
Zsh最大のイケテなさがこれで解消できそうです。
便利すぎて涙がでそう。(自分で解決できなかったので)

It is wonderful (in Japanese as I was convinced by the Japanese).
It seems that Zsh's greatest annoyance can be solved with this.
Thank you.

超コメント遅くてすいません、お役に立てて良かったです!

Thanks, but under what license is this file?

I decided to put it under MIT: https://gist.github.com/xkikeg/3802dfea076c77c880f15229dd909220

export does not work with non-default file path:

> zshhist.py export "${ZDOTDIR}/history" > history.txt
Traceback (most recent call last):
  File "/home/x/.local/bin/zshhist.py", line 136, in <module>
    main()
  File "/home/x/.local/bin/zshhist.py", line 131, in main
    args.func(args)
  File "/home/x/.local/bin/zshhist.py", line 97, in handle_export
    hists = mapNotNone(f, enumerate(args.source.readlines()))
                                    ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'readlines'

Also, no help or description how to use this. Had to read code to understand, but most users probably can't do that.

@shvchk I've tried some examples such as

% ./zshhist.py export /foo/bar > /dev/null
% ./zshhist.py export /path/do/not/exists > /dev/null
zshhist.py export: error: argument source: can't open '/path/do/not/exists': [Errno 2] No such file or directory: '/path/do/not/exists'
% ./zshhist.py export ~/local/ > /dev/null
zshhist.py export: error: argument source: can't open '/home/kikeg/local/': [Errno 21] Is a directory: '/home/kikeg/local/'

but all worked as expected. can you share Python version with python3 --version? Mine is 3.9.2.

For usage, help would appear if you execute it without args.

 % python3 zshhist.py
usage: zshhist.py [-h] {export,import} ...

zsh history editor

positional arguments:
  {export,import}
    export         export zsh history file to UTF-8 text.
    import         transform UTF-8 text to zsh history format.

optional arguments:
  -h, --help       show this help message and exit

Currently my expectation is user can see the doc comments above and find how to use.

(Note: this tool is literally cat if your zsh history won't have non-ASCII UTF-8 chars such as Japanese / Chinese.)

Btw in default=open(DEFAULT_ZSH_HIST_FILE, "rb"), you try to open a file that may not exist, but that may not be the error encountered by the user

@xkikeg thank you for adding help and description! Works as expected if I comment out default=open(DEFAULT_ZSH_HIST_FILE, "rb") line : ) Looks like open() is triggered no matter what. I don't have .zsh_history file in my home dir.

> python3 --version
Python 3.11.4

At first I thought that something might be wrong with my env, so I tested in a container, it's reproducible (replace with docker if you prefer it, command is the same):

podman run --rm -it -w /tmp debian

Then in the container:

apt update; apt install python3 wget -y
wget https://gist.github.com/xkikeg/4162343/raw/zshhist.py
python3 zshhist.py

@shvchk Thanks Andrei for debugging! that's the exact root cause, default args are always evaluated. I've updated not to open it if the arg is specified.

many thx to you~

why zsh returns the commands from history in the raw state rather than decoded to normal UTF if zsh has such intentional way to store the UTF?