About two weeks ago, I ran 9 users studies to see what it was like getting started with F# if you don’t have any F# or functional programming experience. The study was pretty simple, with this as the prompt:
Write a console application in F# that accepts an array of strings as input. Reverse each of the strings in that array. There are no requirements around tools to use or what to search for. The only requirement is that you use F# to accomplish this task.
After prompting them, I just sat and watched as they attempted to complete the task.
Going in, I had a few hypotheses:
People with no F# or functional programming experience…
- …will not be able to complete the task in under an hour because the paradigm shift is too difficult to handle
- …will not be able to complete the task in under an hour because there is not ample information online to figure out how to solve the problem
- …will not be able to complete the task in under an hour because F#-specific rules are too difficult to understand and work with
- …will not be able to complete the task in under an hour because the tooling is too difficult to use or install
Summary
Here’s how it looked w.r.t. the hypotheses:
[Invalidated]…will not be able to complete the task in under an hour because the paradigm shift is too difficult to handle
[Invalidated]…will not be able to complete the task in under an hour because there is not ample information online to figure out how to solve the problem
[validated]…will not be able to complete the task in under an hour because F#-specific rules are too difficult to understand and work with
[validated for VSCode]…will not be able to complete the task in under an hour because the tooling is too difficult to use or install
I learned some more specific things about the experience that were not captured in the hypotheses:
- Between StackOverflow, docs.microsoft.com, dot.net tutorial, and F# for Fun and Profit, there are a significant amount of resources for people to learn how to do something with F#
- There is a separate hypothesis that there may be too much and that might confuse people, but I didn’t see that really transpire
- Tutorialspoint was used by most people at one point. It contains some outdated information and isn’t all that great compared to the previously-mentioned resources, but they must have good SEO because they popped up near the top of most searches people did
- VSCode was preferred by 4/9 participants, VS was preferred by 5/9
- The 5 who preferred VS were actively developing using VS for their job
- The 4 who preferred VSCode were not actively developing using VS for their job
- There were no issues creating new F# projects and “getting to code”
- Those who used VS were able to easily find F# in the New Project Dialog
- The “flow” of using a command-line -> invoking
dotnet new console -lang F#-> invokingcode .to open VSCode -> installing Ionide to get F# tooling was also painless - Scaffolding a new project with the Ionide plugin in VSCode was also painless
Additionally, 2/9 participants were able to accomplish the task in the allotted time. I expected nobody to do it, so that was a pleasant surprise!
F#-specific rules are too difficult to understand and work with
The most common struggle people had was understanding that functions need to be defined before they are used in F#. All 9/9 participants attempted to define a function after the main function in code, leading to a compile error that was difficult to diagnose. Some people never got past this point. Those who did struggled quite a bit, but eventually figured out that you need to define a function before it’s used by looking at a bunch of examples of F# code online.
Some F# error messages were too difficult for people to understand. For example, one participant could not move past an error that was actually an parsing ambiguity, but the error message merely said, “value is not a function and cannot be applied”. Not only was this jargon too difficult for the participant to handle, it was also being emitted in a somewhat unexpected place. The fix was to add a single whitespace character between a mathematical symbol and a numeric literal.
The good news is that community-driven compile error improvements did show up, and none of these improved error messages led to participants getting stuck. The hypothesis derived from this is that focusing on these error message improvements can help onboarding F# developers.
The tooling is too difficult to use or install
5/9 participants who chose Visual Studio did not have any issues with Visual Studio tooling in the study. They were able to find F# templates and focus primarily on coding the task at hand.
Out of the 4/9 participants who uses VSCode, one succeeded by ignoring the built-in project scaffolding that the Ionide project offers. The 3 participants who used the built-in scaffolding all ended up failing due to numerous problems:
- The artifacts scaffolded were too complex; a “all batteries included” template that included Paket and a FAKE build script proved too much for 1 participant to handle
- The projects scaffolded use .NET Framework, not .NET Core, but Ionide pops up a notification suggesting to analyze projects using the .NET Core-based language service
- This results in project parsing failures because the language service doesn’t understand .NET Framework projects
- These failures caused 1 participant to never move past the error message, as they attempted to chase that down for the remainder of the study session
- No tooling support pops up when projects fail to parse led to another participant to not write much code because the tooling was “broken”
The good news here is that the one participant who just used the .NET CLI to scaffold a project didn’t suffer any of the previously-mentioned tooling problems, needing only to struggle with F#-specific rules 🙃
What we can address
There are some concrete things learned that we could do to address some of the problems people experienced:
- Adjust all F# templates to define a function before the main function
- The hypothesis here is that users are more likely to define their own functions above the main function after seeing a function that is defined in this way
- Adjust the dot.net F# tutorial to show that functions need to be defined before use, asking users to define a new function in their code as such
- Change the Get Started with F# on VSCode tutorial to no longer show how to scaffold projects using the built-in tools, instead deferring to the .NET CLI for project creation
- Work with the Ionide maintainers to have Ionide focus more heavily on .NET Core
Would love to get opinions on this from others who are quite passionate about the getting started space: @isaacabraham @Krzysztof-Cieslak @forki @Rickasaurus @auduchinok @alfonsogarciacaro @ReedCopsey @matthid @Tarmil
Note that other alternative approaches to getting started (e.g., tryfsharp, bolero, etc.) were not explored.