Bringing Bash's powerful command line editing to Microsoft Windows' cmd.exe
code.google.comDo yourself a favor, learn PowerShell if you have to work on Windows. http://technet.microsoft.com/library/bb978526.aspx
While powershell is a valuable and interesting option, the problem with it is that it changes the basic metaphor.
For 40 years unix shells and their descendants and derivatives including cmd.exe have used files and text streams as the metaphor for interconnecting processes. Powershell changes that, and it means the output of one command goes to the input of another command as "objects".
This can be powerful, but it is also very disorienting. Which means it can be hard to learn to do even basic things in powershell, things that would take only a pipe or two and a couple unxutls programs in cmd.exe.
In cmd.exe, easy things are easy and hard things can be really hard. In powershell, hard things are hard (as opposed to "really hard") and easy things are hard.
I'm not a fan of powershell but I have to use it in my line of work.
The only thing that I find sucks is that the pipeline is slow as snails. For example, an svnadmin dump piped to a file which takes 8 mins in cmd.exe takes 14 hours in powershell...
Apart from that it's bearable!
The only thing that I find sucks is that the pipeline is slow as snails.
This is because CreateProcess in Windows is slow. It's the reason that run make on Cygwin on Windows for not-that-large Makefiles is really, really slow. The same Makefile on UNIX and Windows differ in startup time by a wide margin. It's really painful to type "make ..." and sit there for 30 seconds on a fast machine.
CreateProcess is only being called once in this case i.e. to spawn svnadmin. The exact script does the following:
The output from svnadmin has a lot of lines. Due to the fact that PS is written on top of the CLR, it reads each line into an immutable string before writing it to a file. So for every line it has to create a new System.String object and as another poster said GC it later. Also as lines are not predictable length it has to buffer them resulting in more overhead.svnadmin dump d:\repo > repo.dumpEffectively where *NIX shells use a fixed size buffer for pipe operations and operate on streams, PS has to convert it to lines first before writing it out.
That doesn't work when you have approximately 25 bytes per line and a 12Gb file which is where the issue is.
I can appreciate the technical explanation as a programmer, but as an end-user of PS: I don't care. It's slow.
For the majority of tasks it's fast enough. There are a few edge cases though.
That's when doing it the Enterprisey way bites you in the ass..
I don't understand your comment. There's nothing enterprisey about it.
I mean that from Microsoft's perspective. They apparently decided to put an object around something as simple and essential to performance as a line buffer. That's when you should have hired a system programmer to do that job.
Don't get me wrong, I actually like their approach in developing an OO-shell - but if it hurts performance that much someone has taken that paradigm too far. It's the typical case of someone with a hammer (OO programmer) trying to approach everything like it were nails.
That's a fair evaluation and one I agree with entirely.
Really 14hours?!
Have you tried to change the way the dump is done?
Well you can only pipe to file as the output is on stdout. We just ran it through cmd.exe instead. 8 mins is fine - it's a 12Gb repo.
I assume it's related to PS converting every line into a system.string.
Followed by massive GC most likely. Thanks for sharing.
perhaps the real solution to the performance issue you mention is to have svnadmin actually write to a target file instead of crossing process boundaries to redirect stdout ?
have you tried :
Start-Process '{PATHTOSUBVERSION}\svnadmin' -argumentlist "dump {PATH_TO_REPOSITORY}" -RedirectStandardOutput c:\temp\repodump.dmp -Wait
more examples here : http://www.youtube.com/watch?v=9sn2L0E5jT8
Never tried that. I will have a go. Thanks for the suggestion.
Unfortunately its very verbose for a simple case which works fine in cmd so I'm not sure that it warrants using that tool
I do only easy things with Powershell and I found it much nicer than classical shells. It's more consistent in the command names and parameter passing. You also need to learn far fewer commands, because Powershell follows the Unix philosophy of small tools that do one thing well much more than Bash+Unix tools. For example if you do "ls" then you get a table where one of the columns is LastWriteTime. Want to sort by that column? "ls | sort lastwritetime". This works without hassle because ls returns a list of objects, and sort sorts a list of objects by a given property, instead of serializing everything to text that would need to be parsed first. I'm sure that Unixes' ls has an option built in to ls to sort by that column, but off the top of my head I have no idea what it is, and many commands that output lists of things do not have that option.
You even get autocompletion across commands: if you type "ls | sort _" where _ is the cursor, then you get a list of properties that you can sort objects returned by ls by (LastAccessTime, LastWriteTime, Extension, etc).
Just one more way that ms produces developers that cant do anything on their own, or outside visual studio.
Just my experience, but most ms developers have been crippled by their tooling. Powershell is no exception.
"Powershell makes it easier to discover commands"
"You're weak and have been crippled by your tools."
o_o You're suffering from Stockholm Syndrome. Upgrade your tools and be free from menial labor.
I disagree. All the time spent finding and learning new tools can be spent getting things done. I use bash, vi, grep, and man and I can get 95% of what I need done with just those.
Good for you. As long as we're making up numbers: I can get 100% of what I need done in 80% of the time as you, simply because I use superior tools that eliminate menial labour. Get out of your bubble.
> it means the output of one command goes to the input of another command as "objects".
Only if both commands support objects. If one of them doesn't then PowerShell just deals with text. I've sent the output of PowerShell commands to perl scripts and GNU utils without any problems. It just worked. I'm still a powershell newbie, and this has made it easy for me to learn it while still using what I know.
Do yourself a favor and don't lock yourself even more into the Windows environment if you, for whatever reason, have to work with it.
May I ask why?
Because lock-in is the only reason Windows is still so heavily entrenched on the market. This isn't even about the merit of Powershell as a program. This is about a tool which is designed to work on Windows (and Windows only[1]), and thus to increase the strength of the chains that bind you to the operating system.
Ever wondered why Microsoft didn't just make a bash-compatible shell? It's not like they couldn't - it's that they don't want. It would make it easier for people to jump ship, after all. And that's something they hope to prevent.
[1]: Yeah, yeah, there's a PS port to Unix land, but no sane human being will seriously use that. It's pretty much just another waste of time to save people from their mistake of investing into a proprietary tool tied to a proprietary environment.
Many a times, Microsoft will retire the older methods for something blazing new, and ruin what application knowledge you had in the process.
These jumps in "technology" are perhaps forward, but also very much backward. I've seen it time and time again with clients who have bought a new computer and had to deal with growing pains (and hatred) of the blue globe, ribbons, and other 'we changed the gui to make it hard to do what you previously did easily'.
At least in the Linux world, your knowledge isn't decimated on an update. There may be depreciations, but those are rarer than MS "we update the world'.
My experience of Microsoft was that they're the most backward compatible source of technology around. Perhaps you could provide specific examples of breakage?
It's funny that this should come up in a thread about PowerShell, a replacement shell Microsoft released 6 years ago which still hasn't supplanted cmd.exe (which itself is based on syntax from the late 70s).
In 1990 I used the MS-DOS 5 Edit tool and QBasic and picked up the shortcuts Ctrl-insert and Shift-insert for copy and paste.
I'm still using them 22 years later. They still work in most Windows applications and all MS applications even though they're not listed. They could easily have taken them out since Ctrl-C and Ctrl-V have been standard in Windows for so long, but they didn't.
I also got quite comfortable with using F2 and F4 to edit my previous DOS command, which still work in Command Prompt. And since Command Prompt now has tab completion and copy and paste, I don't really feel the need to add a bash-style command history. In fact I can never remember the bash equivalent of some of the cmd history shortcuts I use.
I have, however, installed the GOW package (see bmatzelle on github) for grep, less, wc etc at the command prompt. grep is so much faster than the MS equivalent "find" it's not funny, e.g. I just searched for a 9-digit number across 13 files containing 4 million lines (158MB) and grep took 1 second; find took 57.
It couldn't replace the command prompt because of security issues. Basically it made it very easy for virus writers to run powerful commands that the OS would blindly run. In response, Microsoft not only made scripts not executable by default, there is also a "signing" thingamajig where a script won't be executable unless it is specifically signed off to run[1], including those written by you.
[1]: http://www.hanselman.com/blog/SigningPowerShellScripts.aspx
It also isn't a drop in replacement for the command prompt. They exist as two separate programs, instead of the Unix general Terminal and shells run inside.
In my experience, Powershell has a lot of potential but is hampered by clunky interfaces and the lack of a true robust security system across the OS. Add on top of that the large malware industry around Windows, Powershell is basically a non-starter as a replacement for the vanilla command prompt.
If we're speaking strictly in regards to UI, then this is not a problem unique to Windows. I mean, would you not agree that Ubuntu's implementation of Unity changed the GUI a considerable amount compared to previous versions?
The difference is that in the world of GNU/Linux, a different desktop environment or window manager is usually just a single package manager command away. You're being free to choose. Microsoft will force their UIs on you, and there's nothing you can do about it (except not updating, which is a workaround, not a solution).
LOL. You show me a Linux machine that runs 15-year old binaries without problems, or source code that old that compiles against a modern machine, or even just source code 2 years old, for that matter. I'm not a big Microsoft fan but saying that they're less backwards compatible than the alternatives is talk from the looney bin.
"At least in the Linux world, your knowledge isn't decimated on an update."
Of course, so that's why the time I put into learning how LILO worked is now paying off so greatly! Oh no wait it doesn't, because LILO went the way of the dodo (as far as I can tell, I don't use Linux all that often any more). Wait, all that time I used to spend on manually insmod'ing kernel modules to get my audio to work with the various audio systems, that has paid off! Hmm no, that too is a crapshoot that changes every year or so. Config file format? Different between all programs. Desktop integration? Different desktop systems, who each change how they work every 2 or 3 years. Gfx card drivers? Depends on the brand, make and (if you're lucky) distro. Networking setup? Lol, don't get me started.
You're only talking about all the trivial stuff, the IT setup, getting drivers to work with different hardware. This is the just the same on windows, except it has a much broader support for more hardware.
Of course things like config files change, but unix remains one of the most backwards compatible platforms to build anything on.
Did you only read a few words of my post? How about binary (or even source code) compatibility with everything that is not POSIX (which is most of what a modern application needs), desktop environments, the very ways that applications interact with the user? (where files and data are stored etc.) What else is there? Sure, man 3 printf may have had the same interface for 30 years, but it's disingenuous to use that to claim that it's "the most backwards compatible platforms to build anything on".
What's even more funny is that you use "Unix" instead of "Linux", implying that software written for one Unix would magically work with others (I mean, that would be case if all was backwards compatible, right?). Have you ever written software for more than one Unix, even disregarding historical versions? The most painful concoctions needed to develop for several Unixy OS's (like, say, autotools) exist for the exact reason that they're not compatible at all. While at the same time, I've written code for Win98 almost 15 years ago whose binaries still run, without as much as changing a software setting, on Windows 7 (and probably 8 although I haven't tried) today. (again, there are plenty of things to critisize Windows for, but backward compatibility is not one of them).
"Of course things like config files change"
Yeah, that's like saying "of course a 2012 BMW is different in a few minor ways from a Model T, but essentially they're just machines with engines and wheels." Eh, yeah, sure.
Except you can't change your default shell to it, and expect your system to work. You also can't expect that it's installed by default.
Powershell is retarded. Microsoft uses it for their "web services" too which defeats the entire fucking point of it being a web service.
Not sure why I'm getting hate for this. If you want to write a client for MSOffice365 you have to be on a Windows machine... and its a web-based "cloud" service. Makes no sense if you ask me.
Is this different from Unix Utils? Every gig I've ever been on where I'm forced to use a Windows box, first thing I do is load the Unix Util exes into some directory and add it to the path. http://unxutils.sourceforge.net/
If you like unixutils, you might like GOW a bit more:
https://github.com/bmatzelle/gow/wiki/executables_list
https://github.com/bmatzelle/gow/wiki
Kept up to date, every .exe (once installed) is self-contained (no external .DLLs) and portable to just about any Windows box I've run into.
No Cygwin or MinGW installation required.
Bash is actually available in the list (albeit an older version) if you really wanted "bash on Windows"
Are you sure it's kept up to date?
The initial github commit has comment "Initial import of the code I developed back in 2006. Nothing has been changed since then.", and nothing has been changed since that commit (with exception of docs, Gow.nsi and gow.vbs files).
Thanks. I used to use GNUWin32 (http://gnuwin32.sourceforge.net/), but this seems like a good alternative.
This was my thought too, but I pushed GOW to a lower priority in my to-do list after a comment in a recent thread that publicised it:
Yes, it's different. UnxUtls gives you grep, tar, tr, gzip, and etc. Clink appears to give you shell command completion and history completion.
They are complementary.
Which is nice, because you get fine grained control over which bandaids you put over cmd.exe. This basically gives you readline and not much else, and you don't have to install multiple possibly overlapping bandaids to get it.
There is also busybox for Windows - https://github.com/pclouds/busybox-w32
Looks sweet.
I currently have to use Windows at the customer I'm working for. I've installed Console which gives tabs and better copy-n-paste on Windows. See http://sourceforge.net/projects/console/
I'm not sure if clink and Console will work together, but I'll have to try it.
Console2 regularly crashes on my Windows 7, and it looks like it's not actively developed.
I'm giving ConEmu a try and it looks awesome!
http://code.google.com/p/conemu-maximus5/
Scott Hanselman wrote a blog post about it:
http://www.hanselman.com/blog/ConEmuTheWindowsTerminalConsol...
Console2 looks releases 1-2 releases per yer. Looks quite stable while not active as some other projects. Check dates here: http://sourceforge.net/projects/console/files/console-devel/...
I've never had a single crash (of Console2) on my Win7.
I have tested and clink works with Console. Actually I don't see reason why it shouldn't work with Console. Maybe there might be shortcuts collisions but I have not met yet any.
I use console too and it's awesome. I love the tabs and the transparency feature.
yea, i've been using Console2 for the last year and absolutely love it.
I've switched to powershell ISE on windows (shipped with it) rather than cmd hosted powershell. Does the job!
Is it worth installing .NET 3.5 to run the ISE? I use Powershell but only have .NET 4 which runs it just fine. I'm a little surprised that ISE requires 3.5.
It doesn't make any difference. It required 3.5 as win7 shipped with that framework. You may actually have 3.5 installed already. Check for the following dir: c:\windows\Microsoft.net\framework\v3.5
If it's there you have it already.
Have you tried the ISE on Win8 yet? It gets some nice upgrades around intellisense.
Not yet - will fire up my vm later. Thanks for the tip!
What's wrong with Powershell?
I love PowerShell dearly, and I miss it on Unix systems, but what this is trying to fix, and what still stinks at the moment, is the command-line editing situation in Console, which has nothing to do with PowerShell v. Bash v. cmd.exe.
as a .NET developer who sits in both Linux and Microsoft worlds, but primarily in Microsoft technologies, I must say that PowerShell is a very useful and amazing addition to Windows. I did have to learn it, but once I did, I love it.
Yet another tool to learn.
It is definitely worth it for some people, but time is a limited resource, and others will benefit from this kind of project.
starts too slow.
on my PC it is instantaneous, it is a matter of hardware.
No, it's not. I have a TI6x laptop, 4gb of RAM, plenty o' CPU and with no other user-space programs running, it still takes FOREVER to start. I feel like i'm launching Eclipse or something.
Try it on a T410, i7 with 8GB of RAM and a SSD, it takes 2s to start.
it's a SHELL for God's sake! I shouldn't need 8GB and SSD!!!
it is not a matter of hardware. On my ex-work desktop (2.8GHz quad core, so fairly quick) it used to take a good 5-10 seconds to be ready from a cold start. Yes, subsequent starts were faster, but it's a flipping shell, it shouldn't need hefty hardware to start quickly!
Absolutely. Even on my work machine (2.2GhZ, 4GB RAM) it loads within a second.
Yeah, I can also handily disagree with this. My test machine has a 6 month old CPU and 6GB of RAM. I can reboot, wait 5 minutes and try to launch PowerShell and still be looking at a 5-15 second wait.
I have the same problem.
just use cygwin
I feel like cygwin manages to combine the worst of UNIX and the worst of Windows. And the commands that use the cygwin runtime are noticeably slower than the native counterparts.
Also, AFAIK there is no 64bit version yet, which for me is a dealbreaker (I often work with files larger than 2GB).
Almost nothing, except in-memory processing, requires 64-bit for dealing with files over 2GB. Piping with utilities is the Unix way, and it works well with files of any size in Cygwin.
Cygwin commands that run slower than Windows counterparts are typically those that are syscall heavy, where those syscalls are significantly different on Windows and need lots of work for emulation. The biggie is fork(); it's better by far to write scripts etc. in such a way that they stream results rather than iterate and create new processes.
So, for example, rather than write a script that converts Unix paths to Windows paths with iterated calls to cygpath -w, instead pipe the paths to cygpath -w -f -. Rather than use pipe-to-sed (like "$(echo $foo | sed 's|bar|baz|')") for ad-hoc edits, try to use shell substitutions instead (like "${foo/bar/baz}").
Another thing that can be slower in Cygwin is find, when run over very large directory trees. I wrote a wrapper script (I call it rdir) that runs "cmd.exe /c dir /b" and massages the output into a Cygwin-style format. I also have the same script written in terms of find, so that my scripts that use it work on Windows, Solaris, Linux and OS X.
But I have to say, the biggest limiting factor in me solving ad-hoc problems is composing the tools available, rather than the actual runtime speed of the tools themselves. Having all the Unix tools available makes my life far easier in this respect. They could be even slower, and I wouldn't mind, because I would still be saving lots of time compared to what Windows provides; and my scripts usually also work on all my other systems running different OSes.
PowerShell doesn't even support simple fork-join like bash does trivially:
I use this idiom a lot when dealing with lots of multi-gigabyte files. PowerShell is mostly useful to me when I need to access Windows-specific stuff that Cygwin doesn't do well, like WMI.for x in {1..10}; do (sleep $x; echo $x) & done; wait> Almost nothing, except in-memory processing, requires 64-bit for dealing with files over 2GB. Piping with utilities is the Unix way, and it works well with files of any size in Cygwin.
Even ls, or wc -c report bogus results with >2GB files. less does not work even if I want to look at just the first few hundreds of lines (and "head -n 1000 | less" is a horrible workaround).
> Cygwin commands that run slower than Windows counterparts are typically those that are syscall heavy
Most unix commands are syscall/filesystem/IO heavy, after all they are file utilities. What you say with find is exactly what I'm talking about. I find that the unix tools ported to Win32 and compiled with mingw are significantly faster.
Eh? What you state about ls, wc and less is directly contrary to my experience. I'm so astonished I created a 30GB test file and tested it:
66 seconds to read 30GB isn't too bad, that's over 400MB/sec. (It's an SSD.) When I said directory listings could be slow, I meant directory listings, not general I/O; simple read() and write() do not need translation (provided you aren't using Cygwin text-mode mount options, which are not recommended).$ cmd /c dir k.txt Volume in drive C is CobraRoot Volume Serial Number is 02D8-502C Directory of C:\Users\barrkel\AppData\Local\Temp 2012-07-01 15:24 31,292,160,000 k.txt 1 File(s) 31,292,160,000 bytes 0 Dir(s) 142,087,471,104 bytes free $ du -h k.txt 30G k.txt $ ls -l k.txt -rw-r--r--+ 1 barrkel None 31292160000 Jul 1 15:24 k.txt $ wc -c k.txt 31292160000 k.txt $ time wc -l k.txt 6400000000 k.txt real 1m5.651s user 0m46.207s sys 0m8.642s
this works just fine; when I do > to go to the end of the file, it goes there immediately, but stays busy calculating line numbers (it's scanning the whole file); if I cancel with Ctrl+C, it stops, just like it does on other Unix OSes.$ less k.txtPS: It's the mingw tools that don't work properly! I tried it a couple of times, but all the incompatibilities made me give up pretty quickly.
Thanks, that's very interesting, maybe I should give cygwin another try. Last time it was a couple of years ago and I had all the mentioned problems, then I decided to wait until a 64bit version before trying it again...
If I had to guess, I'd say somehow you ended up with text-mode mounts in your previous experience. The default, and recommended, is binary mode, but you're given a choice on install. It affects C programs that specify "t" to fopen() and friends, and causes Cygwin to convert line endings to and from DOS. But it's more trouble than it's worth.
I'm with you there -- I find Cygwin starts slow and unstable, and then seems to rot at an astonishing speed, to the point where it's usually unusable after I've had it installed for a couple of months.
Cygwin doesn't rot. It doesn't automatically update, and nor does it self-configure, so there's nothing to cause the rot. I've never had problems like you describe.
The biggest problem - and what I suspect is happening to you - is when you have third-party programs and utilities that interfere with Cygwin, most often by putting an older or newer version of cygwin1.dll on the $PATH (i.e. you may be using Cygwin as part of some other program and not be aware of it). Cygwin uses shared memory; multiple versions of cygwin1.dll disagree on the format of this shared memory, and things go pear-shaped pretty quickly from there.
Also some antivirus programs can trip up Cygwin; in its emulation, it sometimes has cause to open, close then open files in quick succession, but AV programs sometimes analyse files when they are opened by programs, and cause bogus timing-dependent sharing errors.
I love Cygwin, but I have experienced what I would describe as rot. (manifested via errors on fork()). I think that's caused by things changing around it, via windows dll updates, but I don't have a lot of insight into windows' dll handling and whatnot so I don't know for sure. Also pretty sure installing new cygwin packages can cause this. Usually, rebaseall fixes such problems.
In any event, I much prefer Cygwin to the alternatives.
"there's nothing to cause the rot"
The rot tends to set in as I install packages to Cygwin; the more I add, the slower and less stable Cygwin seems to become, to the point of taking tens of seconds to reach a prompt after opening. Make of that what you will.
"Also some antivirus programs can trip up Cygwin"
That could well be a contributing factor in my case.
Did you try disabling bash completion[1]?
[1] http://cfc.kizzx2.com/index.php/tag/cygwin-slow-performance-...
No, I didn't -- good spot, thank you.
I haven't bothered re-installing cygwin since the last nuke-and-pave, but I'll bear that in mind if I do and it's still relevant.
Windows lacks fork(), or at least it's not documented. To implement the Cygwin folks have to go through a lot of tricks to implement it. And one of them is rebasing all dlls to start at different addresses.
There is a tool to rebase everything, and it's usually started after install. It's possible that after you have recompiled your own apps/dlls they might need rebasing too (speaking as a cygwin user, not developer).
The only thing that I found annoying in cygwin, is that I can't use all commands from the cmd.exe, because they make sense only under cygwin. For example "gcc" is just a cygwin symbolic link to "gcc-something.exe", and when you "run it" it gives "Access denied". The workaround is to run like this: sh -c "gcc <args>" from cmd.exe
But it might solve very hard problems, for example - redis (from antirez's depot) compiled with cygwin worked for me, and although there is much better windows version by the MSOpenTech guys, it just shows that sometimes it might be the only reasonable way. The other app that comes in mind is rsync.
> Windows lacks fork(), or at least it's not documented.
The NT kernel supports forking, but the Win32 subsystem does not. Because Interix lives outside the Win32 subsystem, it can provide a proper fork() implementation, whereas Cygwin has to live with its somewhat brittle emulation.
Maybe I misunderstand this, but couldn't someone write a driver for Cygwin to have a better fork implementation?
As I understand it, the problem lies with making the different subsystems interact: Forking itself is reasonably easy using NtCreateProcess(), but the Win32 subsystem won't know how to deal with the forked process and stuff will break, including console output.
I don't see Microsoft adding forking support to the Win32 subsystem any time soon, so you'd end up rewriting Cygwin from scratch by reverse engineering Interix...
When I sometimes have to work in Windows (which really makes no sense at those companies since the products I build 99% of the time run on Linux); Cygwin makes Windows usable.
Or just use Far Manager - http://www.farmanager.com - it's Midnight Commander, Norton Commander type of program.
I've watched numerous Russian colleagues use it and it does not look appealing. These same colleagues do not seem to understand the power of a real *nix shell. You end up doing less typing if you really know your shell and how to pipe etc.
You can run cygwin shell from FAR, or any other shell. Seriously.
Think of Far Manager as Midnight Commander, more or less.
Why use this over a cross-compiled version of Bash?