How to programatically call a .bat file on Windows

bat_exec.md

How to programatically call a .bat file on Windows

Using Perl 6 on Windows I want to programatically call a .bat file. Both the path to the .bat file and one of its arguments have a space in it. I can not call the .bat file directly for reasons given at the end of the question. The call I want to make is:

"C:\data\p6 repos\rakudo\perl6-m.bat" --target=mbc "--output=C:/data/p6 repos/rakudo/lib/.precomp/EA6F1A9F9A17B17B899C4C7C4A775A883DAF537E/33/33A52796DB3EBB40BEF94B7696A1B0AB7A29B5C5.bc" "C:/data/p6 repos/rakudo/lib/CompUnit/Repository/Staging.pm6"

I need to capture its STDIN and STDOUT and retrieve the exit code.

How do I do this?

What I tried to do

According to this superuser answer one has to use the following command to run this (notice the extra quotes at the very start and end of my command):

cmd.exe /C ""C:\data\p6 repos\rakudo\perl6-m.bat" --target=mbc "--output=C:/data/p6 repos/rakudo/lib/.precomp/EA6F1A9F9A17B17B899C4C7C4A775A883DAF537E/33/33A52796DB3EBB40BEF94B7696A1B0AB7A29B5C5.bc" "C:/data/p6 repos/rakudo/lib/CompUnit/Repository/Staging.pm6""

That works when pasted into a cmd command window.

Doing this in Perl 6 errors out. Doing the same in Python gives the same errors. This is my code.

my $perl6 = 'C:\\data\\p6 repos\\rakudo\\perl6-m.bat';
my $bc = 'C:/data/p6 repos/rakudo/lib/.precomp/EA6F1A9F9A17B17B899C4C7C4A775A883DAF537E/33/33A52796DB3EBB40BEF94B7696A1B0AB7A29B5C5.bc';
my $path = 'C:/data/p6 repos/rakudo/lib/CompUnit/Repository/Staging.pm6';

react {
    my $proc = Proc::Async.new(
        'C:\WINDOWS\system32\cmd.exe', '/C', "\"\"$perl6\" --target=mbc \"--output=$bc\" \"$path\"\""
    );

    whenever $proc.stdout {
        say "Out: $_"
    }
    whenever $proc.stderr {
        say "Err: $_"
    }
    whenever $proc.start(ENV => %(RAKUDO_MODULE_DEBUG => 1)) {
        say "Status: {$_.exitcode}"
    }
}

This results in:

Err: The network path was not found.

Using forward slashes in the $perl6 command results in:

Err: The filename, directory name, or volume label syntax is incorrect.

Leaving the surrounding quotation marks of the command off

'C:\WINDOWS\system32\cmd.exe', '/C', "\"$perl6\" --target=mbc \"--output=$bc\" \"$path\""

results in:

Err: '\"C:\data\p6 repos\rakudo\perl6-m.bat\"' is not recognized as an internal or external command,
operable program or batch file
Err: .

Passing the arguments separately in the command

'C:\WINDOWS\system32\cmd.exe', '/C', $perl6, "--target=mbc", "--output=$bc", $path

results in:

Err: 'C:\data\p6' is not recognized as an internal or external command,
operable program or batch file.

Quoting the executable but leaving the arguments separately

'C:\WINDOWS\system32\cmd.exe', '/C', "\"$perl6\"", "--target=mbc", "--output=$bc", $path

results in:

Err: The filename, directory name, or volume label syntax is incorrect.

Also quoting the other arguments separately

'C:\WINDOWS\system32\cmd.exe', '/C', "\"$perl6\"", "\"--target=mbc\"", "\"--output=$bc\"", "\"$path\""

results in:

Err: '\"C:\data\p6 repos\rakudo\perl6-m.bat\"" "\"--target=mbc\"" "\"--output=C:/data/p6 repos/rakudo/lib/.precomp/EA6F1A9F9A17B17B8
Err: 99C4C7C4A775A883DAF537E/33/33A52796DB3EBB40BEF94B7696A1B0AB7A29B5C5.bc\"" "\"C:/data/p6 repos/rakudo/lib/CompUnit/Repository/Staging.pm6\"' is not recognized as an internal or external command,
operable program or batch file.

Keeping the arguments separate but adding quotationmarks at the beginning and end

'C:\WINDOWS\system32\cmd.exe', '/C', "\"\"$perl6\"", "\"--target=mbc\"", "\"--output=$bc\"", "\"$path\"\""

results in:

Err: The network path was not found.

The errors I get are largely the same when using comparable code on Python 3 and nodejs. So I'm pretty sure this is a problem further down.

So I think I tried everything one can possibly think of. I start to suspect that's a bug in cmd.exe and it's impossible to do.

What node.js does

On node.js the following works:

complete = '""C:\\data\\p6 repos\\rakudo\\perl6-m.bat" --target=mbc "--output=C:/data/p6 repos/rakudo/lib/.precomp/EA6F1A9F9A17B17B899C4C7C4A775A883DAF537E/33/33A52796DB3EBB40BEF94B7696A1B0AB7A29B5C5.bc" "C:/data/p6 repos/rakudo/lib/CompUnit/Repository/Staging.pm6""'

const { spawn } = require('child_process');
const bat = spawn('cmd.exe', ['/c', complete], { shell: true });

bat.stdout.on('data', (data) => {
  console.log(data.toString());
});

bat.stderr.on('data', (data) => {
  console.log(data.toString());
});

bat.on('exit', (code) => {
  console.log(`Child exited with code ${code}`);
});

The trick is, that the above command results in two wrapped cmd.exe calls. That in turn gets rid of the limitation described above.

Sadly just copying this trick in Perl 6 still doesn't succeed

'C:\WINDOWS\system32\cmd.exe', '/d', '/s', '/c', "\"cmd.exe /c \"\"$perl6\" --target=mbc \"--output=$bc\" \"$path\"\""

That results in:

Err: The network path was not found.

The reason is, that libuv messes with the arguments it gets passed and puts a backslash before every quotation mark. There is a flag to turn that off called UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS. The reason this works on node.js is, that it has some special handling that is triggered when 1. on windows, 2. the shell argument is passed to spawn and 3. the command one executes is cmd.exe again. In this very specific situation node passes the options /d /s /c to its cmd.exe and activates the UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS flag.

When adding that flag to moarvms implementation of spawnprocasync it works in Perl 6 also. Though I have no idea what other side effects that will have.

Why I can't call the .bat file directly

When calling the .bat file directly the respective line in the Perl 6 code above becomes

 my $proc = Proc::Async.new( $perl6, "--target=mbc", "--output=$bc", $path );

The code errors out with:

Err: 'C:/data/p6' is not recognized as an internal or external command, operable program or batch file.

Here is some similar code in Python 3.

import subprocess

perl6 = 'C:/data/p6 repos/rakudo/perl6-m.bat'
bc = 'C:/data/p6 repos/rakudo/lib/.precomp/EA6F1A9F9A17B17B899C4C7C4A775A883DAF537E/33/33A52796DB3EBB40BEF94B7696A1B0AB7A29B5C5.bc'
path = 'C:/data/p6 repos/rakudo/lib/CompUnit/Repository/Staging.pm6'

p = subprocess.Popen([perl6, "--target=mbc", f"--output={bc}", path], stdout=subprocess.PIPE)
while p.poll() is None:
    l = p.stdout.readline()
    print(l)
print(p.stdout.read())

On Python this works.

The reason is, that Python calls the Windows API function CreateProcessW without specifying the first argument lpApplicationName. This is the place the actual call happens. application_name is left empty as long as one doesn't pass the executable argument to the subprocesss.Popen call above. When one passes the executable argument to subprocess.Popen the error is the same I get using Perl 6.

p = subprocess.Popen([perl6, "--target=mbc", f"--output={bc}", path], executable="\"C:/data/p6 repos/rakudo/perl6-m.bat\"", stdout=subprocess.PIPE)

Perl 6 uses libuv under the hood. In libuv it is currently not possible to not pass the first argument to CreateProcessW. Thus I can not call the .bat file directly as is possible using Python. The node.js docs (node.js also uses libuv) tell the same story.

commandline-processing.md

Command line processing on Windows

Do read this awesome writeup of how commandline processing works on Windows.

Also don't miss this bugreport that already has a proposal of how to handle this in Perl 6.

Windows does not have the concept of an ARGV array. At the API level there is only one single string that is passed. There is a convention (CommandLineToArgv() implements that) of how a string array is converted to a single string and back. With rare exceptions all programs adhere to this convention. The most prominent exception is cmd.exe. cmd.exe just directly processes its argument string as if you'd have pasted it into a cmd.exe window. When calling a .bat file, Windows implicitly wraps the call in cmd.exe /C <your.bat stuff>. (Not entirely sure what exact command Windows actually puts around the command, it's difficult to find out.) Thus calling any .bat file is subject to cmd.exe processing. One usually doesn't want that.

When calling an .exe file through cmd.exe, then cmd.exe will process the argument string first and the output is afterwards dequoted using CommandLineToArgv(). When calling a .bat file with cmd.exe (irrespective of implicitly or explicitly), then CommandLineToArgv() will not be called. The arguments will arrive in the .bat file as cmd.exe processed them.

The CommandLineToArgv() quoting

If the argument contains ' \n\t\v"' it needs quotation as follows. Put it in "" and put a \ before every " the argument contains. If a contained " is preceded by one or more , duplicate all the . Also duplicate all \ at the end of the argument. One must not duplicate any other . All the so quoted and unquoted arguments are joined with a single space in between.

The cmd.exe quoting

• Either: Prefix all cmd.exe metacharacters (those are ( ) % ! ^ " < > & |) with the escape character ^. It's not possible to escape spaces using ^.
• Or: Wrap arguments in " and prefix all contained " with ^

Five possible usage scenarios

In decreasing order of commonness.

1. I want to run a exe file
   • Then I don't use cmd.exe
   • I want CommandLineToArgv() compatible quoting
2. I want to run a bat file
   • I don't want any cmd.exe side effects
   • I thus need shell quoting, but no CommandLineToArgv() quoting
3. I want to do shell stuff (or use one of the other programs that have different argument parsing, e.g. nmake)
   • Irrespective whether I want to start a .exe or .bat or something else altogether: I want the side effects cmd has
   • No quoting at all. Otherwise I'd circumvent the side effects of cmd
4. I want to do something bare metal. I know what i do:
   • No quoting at all
5. I want to run a exe file via cmd, but don't want any cmd side effects
   • I'd want CommandLineToArgv() quoting first, then shell quoting on the result
   • That's what the Microsoft blog recommends.
   • Why would one want to do such a thing and not just call the exe file directly?

Number 4. and 5. won't happen in any normal usage scenarios.

What others do

libuv quotes arguments compatible to CommandLineToArgv(). When passing the UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS argument this quoting is disabled and the arguments are just joined with a space.

node exposes a shell flag which wraps the call in cmd.exe /d /s /c "<command>" by hand and enables UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS.

javas ProcessBuilder has a very sophisticated and complex processing of arguments. e.g. it detects whether the user called a .bat file or .cmd file and acts on that.

node exposes windowsVerbatimArguments: true when spawning processes. It's certainly pragmatic, but how well does that map to future backends? To that end I wonder if a good (but complicated) solution is something like quotemeta (but for this specific purpose) living inside rakudo such that users themselves would call this on whatever argument instead of passing e.g. windowsVerbatimArguments: true to run/shell. Of course from a backwards compatibility perspective that wouldn't work since UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS would need to be disabled in MoarVM.

@ugexe: Can you elaborate a bit more on the quotemeta solution? I think I might misunderstand your idea.

If I understand that idea correctly you propose to just never quote commands in a meaningful way on Windows and leave it completely up to the user. That would have the following effect: Multiple arguments can be passed to Proc::Async, giving the impression they are actually somehow processed separately. But in reality they are just joined together with a space. This counter intuitive behavior would bite everyone trying to do usecase 1. which is by far the largest usecase.

I do like the idea of implementing the quoting in rakudo though. All backends can then just call CreateProcessW directly in the spawnprocasync OP implementation.