The promise of data is yuge; everyone wants to be data-driven. While data will not solve all of your problems, data will help your team and product. Still, many teams struggle with building out their data teams. They get lost in the noise and promises of different hot technologies and products.
In this blog post, we introduce the concept of prefabricated data science (PFDS), an idea we are very excited about at Magniv, and how it can solve a lot of the pain in creating data teams and working with data.
Press enter or click to view image in full size
Prefabricated data science takes inspiration from the rise of web development tools like Render, Railway, and Netlify. PFDS aims to build a data stack out of services and SaaS products to abstract away deep software engineering tasks like infrastructure, DevOps, and compute optimization. PFDS allows companies to build operational data teams in the quickest amount of time.
We will use two example scenarios to illustrate the concept of PFDS.
The first scenario is one we experienced internally at Magniv. We wanted to compile a list of data scientists’ GitHub profiles for outreach campaigns and other internal uses. We decided the best way to build this targeted list is to fetch GitHub stargazers, people who show interest in a repository, by giving it a star. We ended up putting the profile data in a Postgres database, but it is easy to see that a more extensive data team would want to put this into a CRM such as Salesforce or other business tools (e.g., Google Ads). While exploring this idea, we ran a few scripts on our local computer. Because of the rate limits on GitHub’s API and the sheer number of repositories and stargazers that we want to scrape, it became clear to us we will either need to keep our computer on for a few days or run it in the cloud. It occurred to us that this is a perfect use case for our product. We can schedule different Magniv tasks to do the scraping, cleaning, and saving for us. After some work, we strung Magniv, Render, and Railway together to run this scraper. If we did want to use a CRM, we could have easily added Census or Hightouch to load the data into our desired destinations. You can read more about how we did this at Magniv on our docs.
Compare this setup to its equivalent three years ago. The stack would have been entirely different. We would probably spin up a handful of EC2 or Digital Ocean boxes and run individual crons on them. Maybe we would have set up Celery or Gearman to orchestrate the jobs if they were more advanced. We would then interface directly with the Salesforce API to send the data to Salesforce. The setup time for this project would take at least an extra two weeks. On top of that, because there would be no easy error handling and troubleshooting, it would constantly be breaking with no easy way to be notified.
For the second scenario, imagine you are at a company that wants to track customers’ sentiments on the company’s brand based on customer support emails and posts on social media platforms. The data science team decides to develop a sentiment score based on a machine learning model and would also like to create a dashboard for stakeholders to track the score and “themes” of complaints. Classically, this would be a big project that would require scraping different sources, training a model, putting a model into production, and then creating a dashboard to display the results. There would be a lot of scaffolding required at each stage of this data application and tons of error checks to ensure it will run smoothly.
Today, you can quickly build this using a combination of Magniv, HuggingFace, Streamlit/Retool, and a few other tools. Magniv allows you to efficiently orchestrate and schedule the tasks to grab the information and pass it through the model. HuggingFace enables you to easily use pre-trained models that you can use as the downstream segment of your machine learning model. HF also has a hosted API, so you do not need to be too concerned about the resources required to run a robust model. Streamlit and Retool allow you to create a dashboard in literally seconds. The prefabricated route enables you to go from idea to proof of concept in less than a week with a small team of data scientists.
Both these scenarios are great examples of the power of PFDS. Instead of having to build solutions from the ground up that are generally erratic and require a lot more time and money, we can use tools that abstract away all negative engineering work. With the saved time, your engineering team is free to focus on real problems.