Big Programming, Small Programming – Glow, the language
fendrich.seThis idea - having locally unrestricted components connected by a restricted, declarative system - seems to keep popping up and never quite taking off. Hopefully sooner or later someone will spin it just right.
Some related projects:
Opis - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.... - pure functions connected by dataflow combinators (with time travel debugging and model checking at the dataflow level)
Bloom - http://www.bloom-lang.net/ - temporal logic for coordination, message passing for communication with stateful endpoints
Flask - http://dash.harvard.edu/handle/1/2797447 - a c-like language for local sensor computations with a haskell dsl for specifying communication patterns across the sensor network
Then of course there is all the FRP work (eg http://www.haskell.org/haskellwiki/Functional_Reactive_Progr... http://docs.racket-lang.org/frtime/)
Despite all the interest, overlog (the predecessor to bloom) is the only language I've seen so far that has any convincing demos (hdfs and hadoop in 8kloc). I'd love to know if anyone is actually using this sort of stuff in production or, if not, why not? What goes wrong when you try to scale this up?
EDIT: I forgot to mention Mozart/Oz, one of the most interesting dataflow languages and one that explicitly talks about layering progressively less and less deterministic systems of concurrency - http://www.mozart-oz.org/features.html
There's also Orc: http://orc.csres.utexas.edu/
Regarding "not taking off": this model is actually very common in the vfx industry. Most high-end content creation tools (Nuke, Maya, Houdini, etc) have a node graph which you assemble in the tool, where the nodes define processing operations & are usually plugins written in c++ or something like it. It's not a language per-se and the semantics are tightly bound to the individual tools rather than being independently specified, but the underlying idea is the same.
There has been some interesting work from Johnathon Edwards @ http://www.subtext-lang.org/
I've had his blog bookmarked for a while but never got around to actually reading it. I'll have to take a look, thanks.
Weird that nobody mentioned spreadsheets until now... spreadsheets are one of the possible data flow implementations.
Probably few people know but Microsoft Excel has some optimizations like "multithreaded recalculation" http://msdn.microsoft.com/en-us/library/office/bb687899.aspx and "high-performance Excel based applications in financial services" http://msdn.microsoft.com/en-us/library/bb887539.aspx
I included extra resources at the end of my articles on the subject: http://blog.databigbang.com/category/egont/
Also http://www.kamaelia.org/Home.html developed by BBC R&D using Python. It seems to have been quiet for a while now though.
From ancient history (although explicitly based on Hoare's CSP) there was Occam for the Transputer.
Found another one.
Quartz Composer?
BizTalk Orchestrations would probably fit. I also remember an old robot programming game.
Thanks for the list of related projects!
It's a nice idea, but the run-length example is kind-of-misguided, even though that's how everyone would implement it on first try.
A language should push you towards a more robust/faster/simpler solution by making it the simple, elegant thing to do. In K, a run length encoder would be something like:
{(x@&~m),'(+/m)@&~m:~':x}
x is the input
m gets 1 where a position in x is the same as the previous position, 0 where it isn't. (apply ': "each-pair" to ~ "match")
Then take the sum of match (+/m) at where (@&) there is no match (~m).
Finally, add that to x where there is no match (x@&~m, literally, "x at where no m"), and zip it (,' meaning concat each). And the curly brances make it into a function of x.
Now, I understand most people would think that this is unreasonable. However, in K it is easier to produce a vectorized solution than an itertive one - and as an added bonus, the bugs tend to be either of the "wrong spec" or the "off by one" error, eliminating whole classes of potential bugs.
And .. the result is usually quick, thanks to vectorizing (and can go on a GPU easily).
A language that promotes item-at-a-time processing is missing a lot.
I do like an expressive high-level encoding like that, but 1) I just needed a silly example to show what pipes are. In practice you would probably use higher-order library functions (or components) like group for the same thing. 2) In my experience, very compressed expressions can be hard to modify in certain directions. Practical stuff like printing, logging or storing progress every X seconds, which is a small modification in a for-loop, can be a hassle.
Does K have any APL roots by chance?
edit: ignore the question, yes it does:
As intriguing as it looks, I certainly wouldn't want to look at that kind of code all they long.
I can understand the sentiment, and the only thing I can say is:
There's a learning curve; it's steep, but it is well worth it - much like Math notation (give me "x^n" any day over "repeat multiplying x by itself n times") or Music notation, once you're used to it, any notation that is less concise seems arbitrarily and needlessly verbose.
(That's actually a reason not to learn K / APL: When you actually grok it, it's hard to take modern software engineering seriously. A reason to grok them is that -- for some problems -- you're going to become much more productive).
Well, after writing my comment, I went online looking for some quick introduction to the language. I'll admit that there are certainly very interesting facets to that language.
What I can say however is that there's a difference between conciseness in writing and productivity in reading.
For instance,
It's very concise but very hard to grasp quickly. The equivalent, in even 10 lines of code, might be way easier /and/ faster to read and understand.(!R)@&{&/x!/:2_!x}'!RThat being said, I really like the functional aspect of it where most of common operations on list were shortened. (I.e. map, reduce, etc.)
Personally, I think the Arc language strikes a really fine balance of verbose versus concise. It's just unfortunate that there's not a more active community and that it lacks the battery to be productive with it.
My understanding of the idea behind most APL style languages is that it's an expert language, in that you need to take the time to invest in it just like a musician would musical notation.
Once you've done that, the theory is that it is actually quicker and easier to grasp than reading "verbose" lines of code.
That said, I've never taken the time to invest in an APL style language, so I can't speak to whether it is actually true or not.
Makes sense. If you spend enough time it can become second nature, like musical sheets. Still, I find that musical notations is way more limited, so it makes sense to have it concise. But let say people were naming some parts of the songs, and referring to it, and you had to jump and "get into the mind" of the person who wrote it.. than maybe it'd be a different story and more verbosity would be better.
APL/J/K code is really like mathematics, something that most programmers aren’t exposed to in the slightest. If you don’t grok it immediately, then you can actually just sit down and equationally work out the author’s exact thinking, in precise terms—and you level up as a side effect, so you don’t need to work out the same patterns every time you see them. That’s nigh impossible in C++!
Take it from someone who has seen both sides: the concision is a good, good thing.
Sounds pretty cool!
Since you're pretty much at design stage, two ideas are worth exploring:
1. Uniqueness typing (re: linear typing for mutable data) is a neat addition. The paper by John van Groningen et al seems the most appropriate: "Exchanging Sources Between Clean and Haskell: A Double-Edged Front End for the Clean Compiler."
2. The Shen lanaguage http://shenlanguage.org/
Clearly, these may not map directly to the features in the final design, but the ideas they cross-pollinate will be very useful.
1. Yes, the linear typing for mutable data is inspired by Clean. 2. I look at Shen from time to time. It looks nice, but I have never gotten around to play with it.
The moment I see an open source repository for this I'm going to contribute the hell out of it.
This need to exist. This has to exist.
If Internet comments were always this encouraging, the world would be a better place :).
This looks absolutely fantastic. Well, the whole visual programming thing leaves me cold, but the rest looks great. I'd be curious to see how data structures are supposed to look like. I assume they're some kind of structs (hopefully not featuring ridiculous name collisions like Haskell).
Visual programming used to leave me completely cold, but with the emergence of tablets and the promise not to make _everything_ visual, I hope to convince people otherwise. If you have not seen Bret Victor's brilliant "Future of programming" http://vimeo.com/71278954 and "Inventing on principle" http://vimeo.com/36579366, I highly recommend it.
I promise not to make the Haskell struct mistake.
I'll try and find time for it. Thanks for the recommendation.
Names, unless namespaced, collide in every language.
Sure, but having a receiver avoids collisions for accessors. I'll illustrate the point below.
In a typical OO language, eg, Python, you'd do:
In Haskell:class Foo(object): def __init__(self, bar): self.bar = bar foo = Foo("bar") # prints 'bar' print foo.bar
Here, you can see that Haskell generates an accessor 'bar' which is a top-level function. Which means that if you can't have two records with a 'bar' field in the same module. You can have a 'bar' field in a different module, but then you need to a qualified import (import qualified OtherModule as O) in order to work around the conflict (forcing you to type ugly O.bar every time you need the 'bar' accessor from OtherModule). So you end up putting prefixes everywhere, something which in my opinion is a design mistake.data Foo = Foo { bar :: String } main = do let foo = Foo "bar" -- prints 'bar' putStrLn (bar foo)So you write `foo.bar` in Python and `O.bar foo` in Haskell, and you find the former satisfactory but the latter an ugly design mistake? Honestly it doesn't even look that much more typing; and (subjectively!) has a little better semantics, e.g. you can say that bar is a function with the type `Foo -> String`.
One advantage I'll admit is that in Haskell you have to actively watch out for ambiguities and refactor colliding types to import as qualified.
> So you write `foo.bar` in Python and `O.bar foo` in Haskell, and you find the former satisfactory but the latter an ugly design mistake? Honestly it doesn't even look that much more typing; and (subjectively!) has a little better semantics, e.g. you can say that bar is a function with the type `Foo -> String`.
O.bar foo doesn't look that bad, does it? Sure. Except when you have nested data structures. And you very often have nested data structures. Not to mention that I'm not about to split my data definitions into 50 different modules when working with a database. You know, these pesky things things which so often have a field called "id"...
So instead of the language doing for you, you get to do "namespacing" yourself by prefixing your fields by an abbreviated version of your data type name.
This is a very fair criticism, and I believe lenses are the solution to this problem, although I'm not advanced enough in Haskell to provide a round counterargument.
Lenses are amazing. However, they don't solve this issue. What they do is let you pack a getter and a setter in a single "data type" and do operation on them. Say you have a School record with a students field (a list of Student records). You can write a one-liner which, given a School, lets you access only those students with a name starting by 'a'. Or even better, lets you return a copy of School where only those aforementioned students get their grades doubled.
However, you still run into the same namespacing issue we talked about, because at the end of the day, lenses are a bit of wonderful magic on top of the existing, crummy record system.
You'd create a Lens like so (with Control.Lens):
Now you have a 'bar' lens:-- You're supposed to prefix by _ due to Template Haskell magic following, but it's still a plain record, namespacing issues and all data Foo = Foo { _bar :: String } -- TH magic making lenses for all fields makeLenses ''Foo-- prints 'bar' putStrLn (foo^.bar) -- creates a copy of 'foo' with 'bar' set to 'barbar' let foo' = foo & bar .~ "barbar"
In reality though, you will end up either defining a namespace (and a separate file!) for every struct you define, or prefixing your accessor functions with the name of the struct. I do the latter, but I'm not happy about it at all...
Unless you're happy giving all your imports single-letter names (why hello, BASIC circa 1970!) it's more like: foo.bar in Python, OtherModule.bar foo in Haskell. That does seem like quite a lot more typing.
I usually see Data.Map to imported as M, Text.ByteString as BS, etc. If you need more than a couple of these, I'd say it is a red flag and that the code needs to be split up.
It's in my experience much easier to avoid accidental namespace clashes with functions than with accessors (though some modules are intended to be imported qualified). If you need to model a real-world problem, you run into record namespace clashes very easily.
In the big picture, visual programming is very natural. It is, in fact,
so natural that we often make flowcharts of how systems work. It would
be better if these flowcharts would actually be the system, rather than
an incomplete, possibly outdated description.
In the details, visual programming is just an inconvenient gimmick.
IMHO. It may change one day.
We use two types of files to program Glow. The detail files, which can
contain anything, and the overview files, which may only contain
constants and how components are connected to each other. An overview
file may not use macros.
Both are editable with any text editor, but the overview file can also
be edited visually as a vectorized flowchart (vectorized to zoom in on
details)
This looks great, and I encourage you to keep working on it and make this a reality. I've thought along the same lines and I think a language like this has real potential.
I'm pretty surprised I haven't seen J. Paul Morrison's "Flow-Based Programming" mentioned anywhere in the post or these comments. I first read this book years ago as a undergrad and it made a big impact on me. It discusses ideas very similar to the ones proposed for Glow. You can read it free online here:
http://www.jpaulmorrison.com/fbp/
You can find links to several existing implementations of flow-based programming on Wikipedia:
The syntax looks very pretty, but why the colons at the end of block header statements? Python does this too, but I see no reason for it. In my opinion that makes it look a bit cluttered, contrast it for example with the (excellent) coffeescript syntax.
Colon + indentation is actually the most natural way to signify a block that I have ever seen. Pretty, non-cluttering and sort of light for the eyes. And it's not without precedent: it resembles how colon is used in text.
I'm surprised Noflo and Subtext haven't been mentioned. They seem like they have very similar goals.
If the syntax ends up being a great match for your ideas, I've got a lot to look forward to! Syntax is a big aspect of how much or little I enjoy working with a language. I'm hoping for something Python-ish in terms of simplicity and readability, but I think there's room to improve upon even that.
I'm very happy to not see much redundant punctuation where formatting makes meaning clear, and to not see keywords where punctuation makes meaning clear. But since mutability is being deemphasized, maybe we can even get a single equals for equality, and give assignment the longer symbol?
I agree that syntax is more important than us "rational" programmers like to let on. The syntax will be similar to Python, but nothing fantastic.
I think switching the usual meaning of = and == will be confusing. O'Caml uses = and == for deep and shallow comparison respectively and <- for assignment. That is a possibility.
The concurrency features strike me as being influenced strongly by Hoare's CSP[1]. The flowchart like interface seems akin to Harel's reactive diagramming formalisms[2].
[2] http://www.scribd.com/doc/167971960/Modeling-Reactive-System...
Yep, it is influenced by CSP. It is implicit in the text, since the primary influences (Pipes, Go, etc) are directly influenced by CSP.
Going to try to address the whole problem here, not just the trying to invent a new language:
from: http://fendrich.se/blog/2013/01/24/pre-programming-mental-si...
> I am, in fact, a grumpy, skeptical, philosophically materialist atheist.
from: http://fendrich.se/blog/2013/04/09/a-better-soylent-good/
> Adding nootropics and stimulants, like the original Soylent guy did, is also interesting. However, to avoid complicating things, I will add this later. I don’t really know which substances, but at least ginkgo biloba.
I think you are taking on too much risk in order to get attention. Your grumpyness might be a result of some other health condition like sleep apnea that you need to take care of before taking on a radical diet that will leave you sick and more overweight. Take the time you would have spent working on Glow instead working on an existing up-and-coming language like Go. And, good luck. I know what its like to feel like you will try anything to fix your life and get relevant. But inventing a new language is not where its at. Tom Hickey who is one of the smartest people I know and one of the best presenters I've seen can't even convince most to use Clojure. It isn't easy.
What's wrong with inventing a new language? Nobody has to use it. Do they? Or is he staking some vast amount of his personal reputation on it? some how?
I'm suggesting the combination of possible health problems combined with a pattern of trying to reinvent everything. I'm familiar with this pattern myself. I'm suggesting not to go down this road, and instead focus those creative efforts on something with more promise.
I also think that there are problems with all other languages we use, but like I said, this is a very difficult road.
I'm confused. Are you his mom or what? Why shouldn't he go down this road?
If you want me to be blunt, I think he is incapable of it, and he is wasting all of our time with a "what if" post about a language that will never exist. Happy now?
That's already a sunk cost to everyone in the thread. Further criticism - that doesn't address the reasons people are enthused, serves as a time-sink for no return in value.
i.e. if you want it to go away from altruistic concern, you're probably better off addressing the flaws in the concept or ignoring it. ^_^;
As far as I know, a language called GlowScript (compiles to js) exists.
The language author might consider changing its name to something unique, like 'StarLight' or something so as to avoid name controversy, and achieve maximum popularity :)
BTW, this project looks great. I'll keep an eye on it... :)
Dammit.. I checked to see that Glow was not taken :). Almost every name, that is not an outright racial slur is already used. Still, Java and Javascript are completely different languages and that has never lead to much confusion..
Not within the tech world, sure, but ask say a recruiter and it's a coin toss.Still, Java and Javascript are completely different languages and that has never lead to much confusion..When faced with a JS problem in a SPA - around 6 years ago - I had a representative from a client say me "wait, I will install the latest JAVA and see if that fixes it". He even chose the jdk to be safe.
You can always disambiguate. "Radioactive Glow" has a nice ring to it.
This looks awesome! I will be interested to see how well you can make the features fit together. I have been really interested in languages that give you tools for reasoning at multiple levels of abstraction.
Have you tried seeing how a full dependent type system could be used in your design?
I plan to sneak in something similar with a capable type system and contracts for stuff that you can't express in the types. That way you could "one day" use a solver like Z3 and start proving or disproving some of the tractable contracts, without having to even change the programs. A bit like LiquidHaskell.
Not exactly dependent types, but practical, I think.
Why not use an existing imperative language to implement the components?
That's what he was going on about with the C FFI. You can use C to implement components.
So yet another flow-based language/framework? NoFlo is another recent one. Component-based is a very common modern game programming approach BTW. Google for "component entity system". It's a very cool hobby project but will not get wide traction I'm afraid...