Marqeta hack week Q4 2018

A few times a year Marqeta sets aside a week for people to explore ideas, try out new technologies and/or build something they want to build. The Q4 2018 Hack Week was a huge success with 22 projects spanning from a pitch for a new type of API to an architecture for streaming data. Let’s check out some highlights!

Crucible

A vision for the future of applications at Marqeta — from monolith to serverless & microservices, from fragmented UX to a single UX supported by a React component library.

Team: Eli Malone-Shkurkin, Jimmy Song, Brent Luna, Elliott Joo, Nick Petru, Eric Hishinuma, Scott Attwood

CustomerVision

A Slack app that pulls a list of opportunities from Salesforce and then sends queries to Elasticsearch in order to display recent API log entries for integrating customers. This provides real-time notifications and a lightweight query interface to the entire company about customers we are actively managing in order to:

• Provide insight into customer engagement
• Outreach to customers
• Detect integration issues
• Display recent activity for a particular customer

As a bonus, this same Elasticsearch/SFDC -> Webserver -> Slack/SFDC/Confluence/JIRA data pipeline can be leveraged to:

• Build out real-time, customer-facing API logs
• Update Salesforce opportunities with milestone timestamps (e.g., first successful Sandbox JIT gateway response)
• Trigger workflows within JIRA
• Update project plans in Smartsheet/Confluence/etc.

Team: Ben Mo, James Moschou, Tristan Reed

Continuous integration / delivery

A showcase of continuous integration development and delivery process using an event system. It embeds the build, validation, and deployment in one development pipeline resulting in various benefits:

• Enabling parallel development flows
• Fast test and validation results
• Eliminate bottlenecks seen in traditional development cycles
• Consistent build and runtime environment

Team: Chuong Do, Dan Nemeth, Stephanie Larios

Marq’s new digs

The use of NLP to provide a search tool on DiVA App (our analytics application) to easily support internal and external users with both answers and alerts regarding their data. Capabilities include:

• Ask a question such as “my program all-time high”, and receive aggregated data on their volume
• Provides a low barrier to entry on scheduling alerts with phrases such as, “email me when my program transaction volume >100000”
• Ability to save questions to a Dashboard Story, allowing you to easily access these insights in data stories containing other relevant info, like visualizations and tables.

Tech: React and MobX front end, that utilizes Python and Node API’s

Team: Brent Luna, Nirmal Thapa, Rob Tan, Eric Hishinuma, David Herberich

Rollover

Grants partners, their employees, and their cardholders, a new layer of control. The Rollover API leverages existing funding behaviors and card perks with more unique advantages previously unavailable.

Using the Rollovers Microservice, a cardholder can set aside money to be used for:

• “Rainy day” savings
• Credit card debt & loan payment
• Charity donations
• Long-term, retirement saving

Team: Berkley Hinton, Shiv Chadha, Jessica Smith, Sarah Dulski, Chris Kimmel, Chandrasekaran Subramaniam

DiVA live!

DiVA Live! is the antidote to data latency! It is a Streaming Data-Processing Framework to perform incremental aggregations on n-dimensions on time series data sourced from Payments Webhook events via SNS. The fact that the data is streamed to the Aggregator in real-time means you get data that is always fresh.

Technology Stack:

• Amazon SNS
• Amazon Kinesis
• Amazon S3
• Amazon DynamoDB
• Redis

Architecture:

Why is this important?

• Enables users to access real-time aggregated data up to the last hour/day/week/month/year instantaneously. For example, the question, How is My Program’s volume doing today? — would return the actual volume the time the question was asked.
• Another example is our fraud scoring engine needing to know a cardholder’s pattern on transactions in the last n seconds/minutes/hours and to compute real-time statistics on those transactions instantaneously.
• Live datasets which are incrementally aggregated means that there are no longer any dependencies on database query languages. This eliminates another significant latency in getting results from your data — no more waiting for SQL to return an answer.
• A simple to use API will allow any user to create pre-aggregated time-series datasets based on any dimensions available in the webhook event payloads.
• The event-based architecture means that events from the past can be re-streamed to reconstruct aggregated data sets easily.

Team: Rob Tan

Meltan

Meltan is the next generation monitoring, charting, and alerting platform for Marqeta. It uses Prometheus, InfluxDB, and Grafana to create dashboards showing real-time Payments Metrics like authorizations, declines, digital wallet token activations, card terminations, chargebacks, etc. The metrics are tied to Payment Networks, Customers, and Event types. This allows us to slice, dice, and chart data in multiple ways, making it very flexible. PromQL is used to define the Grafana panels, and alerts are set at the panel. This is already in use by teams across Marqeta to monitor various metrics in real time, and take actions wherever needed.

The infrastructure is hosted in AWS. The payment systems and other microservices send events to SNS, which are then fanned out to an SQS queue. Meltan pulls the events from SQS, and exposes them for Prometheus to scrape on an HTTP endpoint. Prometheus is configured to use InfluxDB as the time series database, which is hosted on an EC2 instance. Grafana sits on top as the dashboarding interface, with Prometheus as the data source. This allows us to write PromQL statements and analyze+chart data using a large number of permutations.

Team: Amiel Montecillo, Manish Pandit

Mother of pearl

We explored the idea of teaching complex concepts through the use of gamification. Our belief was that by decomposing the end to end prototyping and integration experience into a series of consumable stages while rewarding the developer when they complete each milestone the process of learning becomes interactive and enjoyable.

For the first version of the game, we asked the developer to review the quick start for the Marqeta API and complete the exercise by providing a card product token to start the game. On a valid card product token, with the required resources (card, funding source), the developer plays the first level where a delivery ship passes through the spaceport.

In future versions we would have additional levels that test the use of card controls, spending limits, FraudStream rules, JIT Gateway configuration, Tokenization and so on.

Team: Raymond Young, Aubrey Browne, Scott Ledbetter, Dilip Ramachandran, Eli Malone-Shkurkin

PMO app (#MakeSimple)

The PMO App is an application to help automate many of the processes that were done manually by the PMO (Project Management Office). Its main features are built on the project workflow from requesting a project, to review, planning, executing, monitoring and closing projects. Instead of having to manually duplicate information in many tools, it allows entering it all into one database and having it published in many places such as Confluence and Slack, thus reducing a lot of the duplication of effort. It also features a Slack Bot that has a number of commands to take action or get data. The application is Ruby on Rails built on top of Airtable, that uses the Confluence and Slack APIs.

Why automation?

• Lots of Manual Effort entering the same content in multiple places. (Google Site, Confluence, Slack, Google Docs)
• Capture data to produce metrics!
• Easier access for the company to get information
• Consistency of project updates

Team: Errol Pinto, Nick Troia, Chris Jackson, Taner Kipfer, Evita Calvin, Jenna Chambers