Playwright for .NET is now stable
github.comPair Playwright with Robot Framework (https://github.com/MarketSquare/robotframework-browser) and you'll get awesome browser automation with solid reporting, good test instrumentation and support for pretty much any TA need you might have besides browser automation.
Playwright is fantastic, but having used a lot Robot Framework in the past, I would not recommend it even to my worst enemy. The experience he continuous from string / to string conversions that you need to do with the content of the steps and the orrible structure of the directories and the amount of glue code to share some common methods around, brought me to don't even take into consideration job offers that have "Robot Framework" in them.
RF does automatic type conversion based on the typehints in the python libraries used. So no need to change types manually. And there is no dictated directory structure. So, stop doing horrible directory structures and the pain goes away. And there is no glue at all needed to share methods at all. Write function in Python/keyword in Robot framework and take it in use using "Resource" or "Library" statement. Done.
test.robot
calc.py*** Settings *** Library calc.py *** Test Cases *** My First Library ${sum} Plus 1 2 Should Be Equal As Integers ${sum} 3
No manual casting, no extra directories, no complicated sharing of functions.def plus(a: float, b: float): return a + b
Oh, I didn't know this existed. Definitely going to try this. I'm currently using Puppeteer Sharp (https://github.com/hardkoded/puppeteer-sharp) and something like:
> await page.GotoAsync("https://playwright.dev/dotnet");
> await page.ScreenshotAsync(new PageScreenshotOptions() { Path = "screenshot.png" });
Is broken, if you have background images in CSS the screenshot happens after Page Load is completed but before all CSS images are loaded so you end up not getting backgrounds in the screenshot. The only solution is to try add a delay before taking the screen grab, and it there's any sort of latency then the delay could result in not getting a good screenshot...
I'm using Playwright right now, migrating horrendous Selenium tests at work and it's a real breeze of fresh air. It just works, it comes battery included with superb debugging tools (step by step execution directly in the browser, trace dumping for further analysis, reliable interactions recorder with pretty decent code generation ...).
It's an amazing piece of software for a domain that was kept on the side of the road for years.
If you already tried Cypress, then it's Cypress but with true multiple browser support, available in 4 languages, without the bloat and a cleaner and more idiomatic API.
Is it less flaky than Cypress? We're constantly running into Cypress bugs, when we find a workaround for one, two more show up.
In my experience yes. We ran into a Cypress bug that was bad enough that we just switched to Playwright and it works great.
As others mentioned, the same guy is involved in Playwright. One thing I wanted to mention is that he is extremely responsive. I posted several issues a number of years ago and he fixed them pronto. And even dug into the underlying Chromium issues.
Yeah he’s good. Helped me with an issue I had. The particular issue I have is actually in puppeteer. But I need a .net library so that’s why I mentioned it.
Not trying to downplay his project cos it’s actually really good.
you need to put this before the Screenshot ?
await page.WaitForLoadStateAsync(LoadState.NetworkIdle);
I’ll give it a go thanks.
Same guy does playwright sharp, he should really have more supporters
OH WOW! I had no idea! Yeah he does deserve more support. I still donno why MS can't contribute to projects directly. But I'm happy there's another possible solution to my problem.
.NET Foundation is where a lot of the contribution back to the community happens, I believe.
Have you tried selinium and phantomjs headless ? I was using that a few years ago and it worked very well.
PhtantomJS was abandoned officially a long time ago because of all these new and better tools.
Selenium is a source of constant bugs and misery - it's truly a waste of time maintaining its usage in a codebase because randomly sometimes tests fail. The C# wrapper for it is even worse, as it does not follow the idioms of C# and is a straight 1:1 port from the Java version. Highlights include getting exceptions from properties when you hover over them during debug in Visual Studio.
it reflects my experience
rerunning tests because always a few of them fail on first run
but I noticed that the biggest reason was that often my browser was a little ahead when it comes to version than engine, and after updating both of them the situation was a kinda better.
Anyway I don't recommend Selenium, it wastes too much time
Selenium is still very flaky. We have almost zero problems with playwright except where we made the mistake. On the other hand, we get all sorts of weird glitches with Selenium and often have to re-run tests for them to pass with no other changes.
I have built selenium test harnesses using webdriverjs and Jest as the runner at my two most recent jobs. Been using the webdriverjs 4 alpha which has/had been in progress forever. I've never had a flakey-ness issue that was the fault of selenium.
I don't understand why it doesn't get more love and support. You can build higher level testing frameworks right on top of it.
I couldn't get it working in an AWS Lambda. May work now but I haven't had time this year to re-visit that project.
Assuming those run a container? With puppeteer on Heroku, I had to install chrome as part of the container, then pass it no-sandbox on startup. I'd guess you'd need something similar with chrome (I'm on mobile, so excuse the formatting):
#Install Chrome for Puppeteer: RUN curl -LO https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb RUN apt-get update -y RUN apt-get install -y ./google-chrome-stable_current_amd64.deb RUN rm google-chrome-stable_current_amd64.deb ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome-stable browser = await Puppeteer.LaunchAsync( new LaunchOptions { //If we're debugging, then open actual chrome: Headless = !Debugger.IsAttached, Args = new string[] { "--no-sandbox" }, });I can run puppeteer in a Lambda fine, it's just that when you navigate to a webpage with a background loaded as a css image it doesn't wait for that to load before taking the screen grab. :(
Could be the same reason why this doesn't work as an Azure Function either:
For those of you wondering at home: Playwright for .NET is the official language port of Playwright, a Node.js based library to automate Chromium, Firefox and WebKit with a single API.
I put that in the original title, but I think it got edited out by a mod. :)
Still requires nodejs though, it is more a wrapper than a port.
It's not really a wrapper - it's a client in client/server architecture - the difference is important because it means that .NET is a first-class citizen here (like selenium and unlike tools like puppeteer-sharp) and that updates propagate to all languages automatically.
The downside is indeed that you need a "server running".
Nice, just saw there is an official Java and Python version too.
So what's the point of it then? Why add more layers of unnecessary abstraction? Why don't just use the node.js library then?
Because for some teams it's easier to write tests in plain C# with NUnit/xUnit libs.
But why? Why shoehorn everything into a single technology? Is it so that certain people never have to learn anything else in their entire life anymore?
Nobody here wants to say it, so I'll just say it and face the downvotes...
JavaScript as a server-side technology sucks. It was cool back in 2010 when NodeJS first came out and event loops were all the rage. But .NET is now 10x faster than NodeJS, statically typed, has better community support, better tooling, broader official libraries, better package manager, and as a bonus it's not JavaScript. The productivity and performance of .NET with Visual Studio is just unmatched.
IMO, the only true part in your comparison is that .NET is faster than nodejs.
Because it's nice to have all tests in a single view and a single language, both unit, integration and end to end.
Also regardless of keeping it in a _single_ language or tool, one might prefer to write tests in a .net language instead of js or python.
This doesn't apply to javascript as much, but might again one day.
But yes, spending my weekends learning a new language and library is not a feature.
I'd much prefer to spend my off time learning more about my target users, how to better manage a team, or the legal regulations surrounding the domain I'm writing code in, or just hitting the gym...
Not to mention as soon as I add a new language to our project stack that means every current dev has to learn a new technology, our build pipeline becomes more complex, we need a second set of coding guidelines, and finding new devs becomes more difficult because they'll need to know 2 languages instead of one or we have to increase the onboarding time for every future dev who roles onto the project.
I think it's uncommon that it's worthwhile to invest in a second language/tech stack on a real project.
Well, a pretty good reason is that C# is statically typed, which a lot of developers prefer. It's arguably a much more powerful language than JavaScript too.
Personally, I work with both C# and JavaScript, and I'd happily use the C# version of Playwright to write and run UI tests.
Typescript adds static type definitions to javascript. Microsoft's own VSCode is written in typescript. Being able to gradually move from one to another as you augment your existing code with type definitions makes it easier to integrate typescript in existing projects.
Well, a pretty good reason is that C# is statically typed, which a lot of developers prefer.
Don't know, everyday I see more and more C# devs preferring Typescript for new projects.
For the frontend of a SPA, sure, I prefer Typescript over JavaScript for that kind of thing.
But for the backend (APIs, services, whatever), I'd personally still go with C# any day - it really is a wonderful language, and the available tooling is fantastic.
So there is some people that move backend apps from C# to Typescript? really?
Why not?
Why do You think that this will somehow limit people ability to learn or even make them worse version of them?
Most of the dev's I know, want to do coding for life: - not learning tools from scratch every few weeks, because of the abandoned library - not fight with npm audit every week - not waiting for 5 minutes for webpack to download internet and build project in CI, while .Net builds are already completed, unit tested and waiting for smoke run.
Projects written in java/c#/python don't have to write their own ad-hoc http api in nodejs to access playwright from their language.
Because the “Enterprise Architect” who hasn’t done any actual coding in 15 years, and still preaches 15-year-old FUD about open source, will be openly hostile to anything that isn’t superficially related to his anointed tech stack. Which will most likely be either .NET or Java.
.Net is now almost fully open sourced, especially the parts that MS recommends starting new projects in.
And Java has always been open source.
Might be easier to set up the backend before testing the front end. You could use your existing models and code to create data in the DB for instance, and then use the web browser to test that things are working.
What's the alternative? Prepare the backend via exclusive frontend operations? That has its own issues like possibly being impossible (e.g. maybe making an admin isn't exposed to the webapp) or being slow, or not being able to create data in a "legacy" state that the current frontend can't do.
Or maybe use C# to create and teardown the data, but then call out to a nodejs process in the middle of the test?
I am legitimately curious how you would test a C# app exclusively from nodejs.
really curious about peoples experience with this compared to things like Cypress.
I've had some issues with Cypress selectors randomly failing to find matching elements, even though they are clearly present. Like if I run the same script 10 times it goes through ~7 times. I wanted to try Playwright for another application to avoid such issues.
And then I was very surprised to get almost the exact same issues with Playwright. Running against a different application with different test cases.
Has anyone else had issues like this with React applications? It's very annoying to not be able to write repeatable tests.
Were you able to use the DOM snapshots to debug why the selectors did not resolve? Would love to know more.
> DOM snapshots
I should definitely try that the next time I'm working on this issue. Maybe I can get some kind of small example as well if possible. But replicating the problematic conditions is probably hard.
Got less once I started using dom-testing-library :)
Well, Cypress is going to use playwright for their Webkit integration.
Cypress is more limited compared to Playwright such as multi-domains aren't supported, multiple tabs, or popup windows. For my tests that require that I am using Playwright and for the rest I am currently using Cypress. But I do think the test writing experience is better with Cypress :)
I am considering to move to Playwright for everything now with the new Trace Viewer (https://playwright.dev/docs/debug#playwright-trace-viewer); and contribute to that.
> such as multi-domains aren't supported
Thankfully this is being actively worked on: https://github.com/cypress-io/cypress/issues/944#issuecommen...
This question does come up quite a lot. One thing the remember is that Cypress is "batteries included" testing framework, where Playwright is essentially just the browser automation part of it.
I know the Playwright team is "moving up the stack" and adding many specific testing features, but it's not at the same level (yet?) as Cypress.
I think you're thinking of Puppeteer? Playwright has stuff like automatically waiting for elements, selecting by text etc all built in - with modern JavaScript/TypeScript, none of the weird chaining syntax and debugger.
There are also tools that give you pretty much any feature Cypress has on top of Playwright.
(Not saying which is better - only that they're both sort of the same slot)
With the latest release, Playwright includes an integrated test runner: https://playwright.dev/docs/next/test-intro
Does anyone know if this can be used for effective performance testing? It’s not clear from the Playwright documentation, but I know Selenium (a similar tool) does not recommend performance testing.
We use both Testim.io and Playwright for performance testing.
I work at Microsoft but at a completely unrelated team and while I know some of the folks who work on playwright I have no official affiliation and my opinions represent my own and not my employer's.
Playwright is great for this sort of thing! All we do is run the same test with two variants (before/after the change) and run a student t-test (well, a welch t-test but close enough).
It's ±50 LoC and as long as you're fine with running the test enough times to get statistical significance Playwright works quite well for this
It depends exactly what you mean by performance testing. The problem with a lot of UI tests is that you might need to add artificial slowness to make the tests more likely to pass. Even waiting for something to appear can take longer than it would in a browser. If you want a basic number to see regression, sure you can use this.
If you want to get a sense of how far the system can scale, you would be better with a proper performance testing framework that can run multiple threads, ideally from multiple locations (to avoid any network limits) and built-in support for accurate timing. Apache Bench is pretty common and relatively easy to setup and use. There is also JMeter and even SaaS services to do it for you.
> The problem with a lot of UI tests is that you might need to add artificial slowness to make the tests more likely to pass.
For most real-world performance tests, you should be adding plenty of delay. The average delay on the web between pages for real users runs around 50 seconds, last I looked (which was a while ago, admittedly).
If your app uses keepalives, or polling, or websockets, running your users really fast is going to make your test less accurate and you may get a false positive.
Cool! I hate cypress so much.
I thought it was a tool for me to use openai to generate play scripts.
What's wrong with canopy?
Next time I want to make my eyes bleed I'll hit up .net