Frustration project: Automate data entry into PeopleSoft with Selenium
github.com#FunFact: I started the Selenium project. My first job out of college was at PeopleSoft. I left PeopleSoft and went "in-house" as the PeopleSoft admin at ThoughtWorks. A few years later, Selenium came out as a side-project for another internal system we needed. But when we were rolling out Selenium and dreaming up "world domination plans", one of my mine was to get a copy of Selenium to ship with every copy of PeopleSoft (supplanting the free copy of SQA Robot that PeopleSoft shipped by default). Anyway, I never really pursued that particular "plan", but here we are.
THANK YOU for giving us Selenium. When I first ran my script and saw the browser pop up start "running on it's own," I was pretty wowed.
Selenium is amazing and saved my sanity when I was fresh out of college in a boring office job with terrible software. Thank you for it!
Thank you for starting selenium. It has helped me in so many ways over the years.
> Selenium came out as a side-project for another internal system we needed.
What kind of system? I had thought that selenium was created to TDD web development front-end.
Another +1 for Selenium. I used it to bag tickets for a show I wanted to see. The seller’s site was terrible and apparently had no protection against ticket spinners so all 5,000 tickets sold out in minutes. Fortunately their resale site also had no protection at all so it was easy to write something to reload the page often and put tickets into my shopping cart as soon as they became available. When that happened it notified me and handed back control for me to enter the credit card details (I was too lazy to automate it all). I’m not proud of this brute force approach but if the reseller’s site wasn’t so awful to start with it wouldn’t be necessary.
"Peoplesoft is a curse on humanity"
I would love to know the story of how this wretched piece of software seems to have embedded itself into nearly every college and university in the United States.
The 4 ways to spend money[1], in this case, spending someone else's money on someone else. Likely a procurement person with no IT knowledge, who never has to deal with the painful solution, but got wined and dined by the salesman.
[1] https://investorjunkie.com/economics/milton-friedman-ways-sp...
No surprise that it is backed by Oracle. The sales people at Oracle are really good at getting technology inept people/businesses into multi year contracts.
Once they started citing the list of features in their hundred slide presentation, the administrator stood no chance.
PeopleSoft was one of the larger players, at least in education, well before Oracle bought them. There's really (or at least there was when I was involved in higher education) only 3 main players at that level. And trust me, each option you had was bad in its own way.
The problem is that these products started out much smaller, and grew to fill the needs, although not necessarily as nicely as you might like. So now when it's time to put out a new bid, there's just the handful of vendors who can check off all the boxes for what the current system does. And integration with other systems is always a pain - even today it often includes csv or tab-delimited files that get sent to ftp sites for batch processing!
All these enterprise systems are crap. Trying using SAP. It's not better.
Believe it or not, Peoplesoft was way better than what it replaced at my school.
There is an entire class of business problems that can be solved using Selenium.
I don’t love that raw http connections are the universal API, but at least it’s a tool in our belts when these companies provide lackluster or zero proper interfaces.
There's a whole class of solutions beyond Selenium that automates not just web apps but all apps. It's going under the name of "Robotic Process Automation" and people really like it because they're not dependent on their software vendor to get basic integrations done.
Love it. I've built a flight search tool in selenium too, and for isolating deals. (I've also used selenium for testing). The biggest problem is the speed and the unexpected (the spinner you experienced, intermittent page load times, asynchronous requests or when elements are stale as a result). But it is possible to do it.
Next you can thread the sucker, so you could have 2-4 PS windows open and enter at a faster speed ;)
Yes, multi-threads of it...that would be scary!
I use selenium similarly to drive a similarly awful interface known as costpoint. I would DEARLY love to know how to use the browser developer tools to figure out what is actually happening under the hood of websites without having to read every single Javascript module. The network tab shows you what is sent and received but it's not easy to figure out how those messages are created and parsed. It would be nice to learn how to debug and step through some of that.
If I could learn the underlying api, I could write a tool that would never change. The ui changes much more frequently than the interface between client and server code. Besides, it would be much faster this way.
If you click on the "initiator" column in the network tab it will jump to the code that created the request. Can also see stack traces, set breakpoints, etc.
Amazing - how did I miss these? Thanks!
If anyone happens to know of a particularly good article on this for non-javascript (but technical) people, please post.
Please clarify--are you looking to understand how browser events work without learning about the DOM and/or Javascript?
I suppose something along the lines of Chrome debugging for programmers who aren't javascript developers...undoubtedly there are some fundamentals you would have to become knowledgeable in, as with any technology.
Other comment about initiators is probably more useful for your use-case but there are also "Event Listener Breakpoints" in the sources tab in chrome dev tools. You can use mouse->click to figure out what most buttons are doing (you may have to step through a couple onclick handlers to get to the one you want).
I'm always interested in clever use cases for Selenium.
It seems like "automating boring stuff using Selenium" should be a bigger category than it is - and use cases that can do it profitably are always fascinating.
RPA (Robotic Process Automation) is a big category of tools that can be summarised as ‘automating boring stuff’. You often see it used for repetitive data entry, e.g.
1. Load excel sheet from X:\...
2. Login to Y
3. For each row navigate to ... then enter data
4. Submit, if there’s a validation error add it to and excel workbook
5. Email exception report to ...
What software is generally used for this? Is there anything available for the average person that doesn’t require spending a small fortune?
Uipath and automation anywhere seem to be big players with community/free starter editions. PWC (who are spearheading the RPA at $work) seem to have decreed that all their staff are going to use uipath worldwide.
There are a handful of open source RPA tools, but I don’t think there’s a breakthrough project yet.
Azure LogicApps are good for these kind of silly integrations, if it has a connector for it then it's a piece of cake.
They have a pretty limited use case though.
Booking concert tickets seems to be a popular one: https://github.com/thomasms/glastoselenium
The fact that you have to update Chromedriver with every Chrome release now doesn't make things easier.
Is that new? I seem to remember never updating the driver even after chrome did it's auto updates and still having my code run.
I use a managed service now so I don't have to sorry about that stuff
Chromedriver used to support three major releases of Chrome at a time, now it supports one: https://chromedriver.chromium.org/downloads This means all selenium scripts break every two months.
maybe it would be cool to create a "webdriver" that supports Puppeteer.
> "automating boring stuff using Selenium"
With an Appendix for the sites that block using Selenium.
Webdriver is for more than just testing! Very cool demonstration.
My biggest headache was in getting the right selectors for the various form fields - some services have shifting IDs and classes (ugh xpath)
When you start using xpath that's 20 parents deep and every line of code scrolls off the page you quickly realize how important it is for your devs to to write good accessible and unique element IDs.
IMHO it's usually better to write a WebExtension instead of using Selenium, since you have more control over the browser, can intercept and modify network requests, can access the DOM from JavaScript while being protected from the page altering the environment, and can ship the code as a browser add-on, while still being able to communicate with native apps.
Only downside is that you can't drive old IE and mobile browsers except for Firefox for Android.
Any tools to do this with native applications too not just browsers? We have been flooded by this RPA companies the past 2 years and I'd love a way to do this with Python. It is actually a potentially very profitable market since the impact of automation is very visible (people see cursors do stuff for them...).
I would love to hear any more you have to say about the genetic algorithms you use to help with scheduling.
Ah yes, the genetric algorithm (GA). Well, in short, scheduling personnel is a very visible, policital, and important job. It affects people's lives, and has to be done farily and correctly. I did my best (by hand) for a few years (scheduling 50ish professors into 200 class sections), and I knew I was doing a bad job, but had no choice. It would take 2 weeks to do, because I could only mentally handle working on such a job for so much in a given day before I had to set it aside. So I came up with a preference form, that each could fill out dealing with things like preferred time of the day to teach, daily schedule compactness, preferred class levels, etc. These preferences could also be weighted (0..1) by importance to each person. So the GA starts out with a long, two-part inital random sequence that consists IDs to all professors followed by IDs to all possible classes. Using crossover and mutation (the usual GA steps), the order of the professor IDs vs the order of the classes (prof 0 gets class, prof 1 gets class 1, etc.) are shuffled by the GA. (There are a lot of strange constraints to this you'd only see in academia, related to pay, and contractual teaching requirements.) A fitness is calculated for each professor's latest schedule, by computing a weighted sum of their preferences, based on the assigned classes pulled from the long GA sequence. It's incredible how well it works. It is done in under an hour, people seem happy, everyone's schedule fitness is minimzed, and each's schedule fitness is explained to them (i.e. you got +3 points because you wanted an 8am class, and got one). Everyone can also see everyone else's fitness so it's a "share the pain" kind of resulting schedule. At times when we go in to tweak things by hand, it is clear how tightly the GA optimized things, as any human intervention we try usually and obviously makes things worse. Bottom line, the GA generates the CSV, which Selenium types into PS. After using the GA, I'm convinced that neither humans (or spreadsheets) should be invovled in scheduling at all. I also think GAs are also pretty incredible.
Fascinating! I am going to try that. I’ve been trying to teach myself SAT solvers and constraint systems to approach the problem at my junior college.
Interest! Could you put the genetic algorithm code on GitHub? Would love to learn from it :)
Selenium leaves trace in HTTP requests, so its use is quite detectable. The authors of PeopleSoft can include Selenium detection in their products, disallowing your automation scripts. Why not use GreaseMonkey instead?
Do you have any more information about this trace in HTTP requests? My understanding was that it was the _browser_ doing the requests; Selenium was just driving the browser.
Yes, here is info: https://stackoverflow.com/questions/33225947/can-a-website-d...
Probably referring to the “user agent”, but this can be easily changed.
add AI on top of that which learns what/where/how to enter (or at least suggests) and you get the modern office automation wave.
Ha! I see your crappy UI in PeopleSoft, with hard to automate edge cases, and I raise you multiple SharePoint instances!
Ante up at your peril in this game :/)
I've managed to automate SharePoint data uploads and downloads using the SOAP API.
I have too, and the OP got Peoplesoft to work. But ... my goodness the effort compared to a proper jason APInis huge and compared to just web scraping HTML. I mean there is a word file someone just uploaded. That's three clicks