Projects – LFAI & Data

16 min read Original article ↗

The LF AI & Data Foundation supports open source projects within artificial intelligence and the data space.

Filter by:

1chipML

1chipML is an open source library for basic numerical crunching and machine learning for microcontrollers.

Acumos AI

Acumos AI is a platform and open source framework that makes it easy to build, share, and deploy AI apps.

Adlik

Adlik is a toolkit for accelerating deep learning inference. The goal of Adlik is to accelerate deep learning inference process both on cloud and embedded environments.

Adversarial Robustness Toolbox

Adversarial Robustness Toolbox (ART) provides tools that enable developers and researchers to evaluate, defend, certify and verify Machine Learning models and applications against the adversarial threats.

AI Explainability 360

AI Explainability 360 is an open source toolkit that can help users better understand the ways that machine learning models predict labels using a wide variety of techniques throughout the AI application lifecycle.

AI Fairness 360

AI Fairness 360 is an extensible open source toolkit that can help users understand and mitigate bias in machine learning models throughout the AI application lifecycle.

Amundsen

Amundsen is a data discovery and metadata engine for improving the productivity of data analysts, data scientists and engineers when interacting with data.

Angel ML

The Angel Project is a high-performance distributed machine learning platform based on Parameter Server, running on YARN and Apache Spark.

Artigraph

Artigraph is a tool to improve the authorship, management, and quality of data.

BeeAI

The mission of the BeeAI project is to develop an open-source framework for building production-ready agents.

BeyondML

BeyondML is a framework for developing sparse neural networks that can perform multiple tasks across multiple data domains.

BITOL

Within the BITOL project, the primary objective is to tackle multiple challenges, such as data normalization, ensuring the relevance of documentation, establishing service-level expectations, simplifying data and tool integration, and promoting a data product-oriented approach.

CLAIMED

CLAIMED (Component Library for AI, Machine Learning, ETL and Data Science) is a runtime and programming language agnostic Data & AI component framework.

Data Prep Kit

The mission of the Data Prep Kit project is to simplify unstructured data preparation for LLM application development.

DataPractices

DataPractices is a “Manifesto for Data Practices,” comprised of values and principles to illustrate the most effective, modern, and ethical approach to data teamwork.

Datashim

Datashim is enabling and accelerating data access for Kubernetes/Openshift workloads in a transparent and declarative way.

DeepCausality

DeepCausality is a hyper-geometric computational causality library that enables fast and deterministic context-aware causal reasoning over complex multi-stage causality models.

DeepRec

DeepRec is a high-performance recommendation deep learning framework based on TensorFlow 1.15, Intel-TensorFlow and NVIDIA-TensorFlow.

Delta

DELTA is a deep learning based end-to-end natural language and speech processing platform.

Delta Lake

Delta Lake is an open source storage layer that brings reliability to data lakes.

DLRover

The mission of the DLRover Project is to make the distributed training of large AI models easy, stable, fast and green through the training of deep learning models on a distributed cluster.

DocArray

DocArray is a library for nested, unstructured, multimodal data in transit.

Docling

The mission of the project is to simplify document processing, to parse diverse formats — including advanced PDF understanding — and to provide seamless integrations with the gen AI ecosystem

Egeria

Egeria is the world’s first open source metadata standard. It provides open APIs, event formats, types and integration logic so organizations can share data management and governance across the entireenterprise without reformatting or restricting the data to a single format, platform, or vendor product.

Egeria Conformance

To ensure both consistency and alignment with the standards driven by Egeria, the Egeria Conformance program is available for vendors to showcase how they are shipping Egeria as part of their offering.

Elastic Deep Learning (EDL)

EDL is an Elastic Deep Learning framework designed to help deep learning cloud service providers to build cluster cloud services using deep learning frameworks such as PaddlePaddle and TensorFlow.

Elyra

Elyra is an open-source low code / no code framework for creating reproducible, scalable and component based data science pipelines.

FATE

FATE (Federated AI Technology Enabler) is the world's first industrial grade federated learning open source framework to enable enterprises and institutions to collaborate on data while protecting data security and privacy.

Feast

Feast is an open source feature store for machine learning. It was developed as a collaboration between Gojek and Google in 2018.

Feathr

Feathr is an enterprise-grade, high-performance feature store.

FlagAI

FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale models.

Flyte

Flyte is a production-grade, declarative, structured and highly scalable cloud-native workflow orchestration platform.

ForestFlow

ForestFlow is a scalable policy-based cloud-native machine learning model server.

Horovod

Horovod makes it easy to take a single-GPU TensorFlow program and successfully train it on many GPUs faster. Horovod also achieved significantly improved GPU resource usage figures.

Intersectional Fairness (ISF)

Intersectional Fairness (ISF) is a bias detection and mitigation technology for addressing intersectional bias, which is caused by the combinations of multiple protected attributes.

IREE

IREE (Intermediate Representation Execution Environment) is a sandbox-stage project of LF AI & Data Foundation.

JanusGraph

JanusGraph is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster.

Kedro

Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code.

Kompute

Kompute is a general purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing use cases.

LakeSoul

LakeSoul is a cloud-native Lakehouse framework developed by DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & batch processing.

Ludwig

Ludwig is an open-source, declarative machine learning framework that makes it easy to define deep learning pipelines with a simple and flexible data-driven configuration system.

Machine Learning eXchange

Machine Learning eXchange (MLX) is a Data and AI Assets Catalog and Execution Engine.

Marquez

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem’s metadata.

Milvus

Milvus is an open-source vector database that is highly flexible, reliable, and blazing fast.

Monocle

AI Observability as a Service. A domain specific tracing + metrics framework for monitoring generative AI applications.

NNStreamer

NNStreamer is a set of Gstreamer plugins that support ease and efficiency for Gstreamer developers adopting neural network models and neural network developers managing neural network pipelines and their filters.

OAAX

OAAX or Open AI Accelerator Exchange, is an open standard designed to simplify the development and deployment of edge AI applications. It provides a unified framework for converting and optimizing machine learning models for execution on various hardware accelerators, or XPUs.

ONNX

ONNX is an open format to represent deep learning models.

Open Model Initiative

The mission of OMI is to support open community development of openly licensed baseline AI models for image, video and audio generation that individuals and organizations can use and augment in their own solutions.

Open Platform for Enterprise AI

The mission of the Open Platform for Enterprise AI (OPEA) Project is to develop an ecosystem orchestration framework to efficiently integrate performant GenAI technologies and workflows leading to quicker GenAI adoption and business value.

OpenBytes

OpenBytes aims to facilitate wider sharing of, and collaboration with, data in the AI community through the promotion of data standards and formats and enabling contributions of data.

OpenDataology

OpenDataology is an open source dataset license compliance analysis project.

OpenDS4All

ODPi’s OpenDS4All enables the creation of educational Data Science programs.

OpenFL

OpenFL is a Python 3 library for federated learning that enables organizations to collaboratively train a model without sharing sensitive information.

OpenLineage

OpenLineage proposes an open standard and API for lineage collection.

Pyro

Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend.

Recommenders

Best practices on recommendation systems.

RosaeNLG

RosaeNLG is an open source project, template-based Natural Language Generation (NLG) automating the production of relatively repetitive texts based on structured input data and textual templates, run by a NLG engine.

RWKV

Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv", from 4 major params: R W K V).

RYOMA

The mission of the Project is to build a general LLM agentic framework designed for data analysis.

SapientML

SapientML Project is a Meta-Learning based AutoML initiative designed to enhance the success of the AI model creation process.

ShaderNN

ShaderNN is a lightweight deep learning inference framework optimized for Convolutional Neural Networks on mobile platforms.

SOAJS

SOAJS is an open source microservices and API management platform.

Sparklyr

Sparklyr is an open-source and modern interface to scale data science and machine learning workflows using Apache Spark™, R, and a rich extension ecosystem.

Substra

Substra is a framework offering distributed orchestration of machine learning tasks among partners while guaranteeing secure and trustless traceability of all operations.

The Open Voice Interoperability Initiative

The Open Voice Network Interoperability Initiative is developing The “Message Envelope,” a universal, open API for voice/chatbot and language model interoperability, analogous to HTTP AND HTML.

The Open Voice Network Trust Mark Initiative

The Open Voice Network Trust Mark Initiative translates ethical principles into action, focusing on conversational AI.

TonY

TonY is a framework to natively run deep learning jobs on Apache Hadoop.

Unity Catalog

Unity Catalog is the industry's only universal catalog for data and AI.

Vortex

The mission of the Project is to develop a performant columnar file format and associated tools for database and data lake storage.

Xtreme1

Xtreme1 is the next generation open source platform for multi-sensory training data.

Adversarial Robustness Toolbox

Adversarial Robustness Toolbox (ART) provides tools that enable developers and researchers to evaluate, defend, certify and verify Machine Learning models and applications against the adversarial threats.

Angel ML

The Angel Project is a high-performance distributed machine learning platform based on Parameter Server, running on YARN and Apache Spark.

DocArray

DocArray is a library for nested, unstructured, multimodal data in transit.

Egeria

Egeria is the world’s first open source metadata standard. It provides open APIs, event formats, types and integration logic so organizations can share data management and governance across the entireenterprise without reformatting or restricting the data to a single format, platform, or vendor product.

Flyte

Flyte is a production-grade, declarative, structured and highly scalable cloud-native workflow orchestration platform.

Horovod

Horovod makes it easy to take a single-GPU TensorFlow program and successfully train it on many GPUs faster. Horovod also achieved significantly improved GPU resource usage figures.

Kedro

Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code.

Marquez

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem’s metadata.

Milvus

Milvus is an open-source vector database that is highly flexible, reliable, and blazing fast.

ONNX

ONNX is an open format to represent deep learning models.

OpenLineage

OpenLineage proposes an open standard and API for lineage collection.

Pyro

Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend.

Adlik

Adlik is a toolkit for accelerating deep learning inference. The goal of Adlik is to accelerate deep learning inference process both on cloud and embedded environments.

AI Explainability 360

AI Explainability 360 is an open source toolkit that can help users better understand the ways that machine learning models predict labels using a wide variety of techniques throughout the AI application lifecycle.

AI Fairness 360

AI Fairness 360 is an extensible open source toolkit that can help users understand and mitigate bias in machine learning models throughout the AI application lifecycle.

Amundsen

Amundsen is a data discovery and metadata engine for improving the productivity of data analysts, data scientists and engineers when interacting with data.

BeeAI

The mission of the BeeAI project is to develop an open-source framework for building production-ready agents.

BITOL

Within the BITOL project, the primary objective is to tackle multiple challenges, such as data normalization, ensuring the relevance of documentation, establishing service-level expectations, simplifying data and tool integration, and promoting a data product-oriented approach.

Data Prep Kit

The mission of the Data Prep Kit project is to simplify unstructured data preparation for LLM application development.

Datashim

Datashim is enabling and accelerating data access for Kubernetes/Openshift workloads in a transparent and declarative way.

Delta Lake

Delta Lake is an open source storage layer that brings reliability to data lakes.

Docling

The mission of the project is to simplify document processing, to parse diverse formats — including advanced PDF understanding — and to provide seamless integrations with the gen AI ecosystem

Elyra

Elyra is an open-source low code / no code framework for creating reproducible, scalable and component based data science pipelines.

FATE

FATE (Federated AI Technology Enabler) is the world's first industrial grade federated learning open source framework to enable enterprises and institutions to collaborate on data while protecting data security and privacy.

Feast

Feast is an open source feature store for machine learning. It was developed as a collaboration between Gojek and Google in 2018.

ForestFlow

ForestFlow is a scalable policy-based cloud-native machine learning model server.

JanusGraph

JanusGraph is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster.

Kompute

Kompute is a general purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing use cases.

Ludwig

Ludwig is an open-source, declarative machine learning framework that makes it easy to define deep learning pipelines with a simple and flexible data-driven configuration system.

NNStreamer

NNStreamer is a set of Gstreamer plugins that support ease and efficiency for Gstreamer developers adopting neural network models and neural network developers managing neural network pipelines and their filters.

OpenDS4All

ODPi’s OpenDS4All enables the creation of educational Data Science programs.

RWKV

Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv", from 4 major params: R W K V).

SOAJS

SOAJS is an open source microservices and API management platform.

Sparklyr

Sparklyr is an open-source and modern interface to scale data science and machine learning workflows using Apache Spark™, R, and a rich extension ecosystem.

Substra

Substra is a framework offering distributed orchestration of machine learning tasks among partners while guaranteeing secure and trustless traceability of all operations.

The Open Voice Interoperability Initiative

The Open Voice Network Interoperability Initiative is developing The “Message Envelope,” a universal, open API for voice/chatbot and language model interoperability, analogous to HTTP AND HTML.

The Open Voice Network Trust Mark Initiative

The Open Voice Network Trust Mark Initiative translates ethical principles into action, focusing on conversational AI.

Vortex

The mission of the Project is to develop a performant columnar file format and associated tools for database and data lake storage.

Artigraph

Artigraph is a tool to improve the authorship, management, and quality of data.

CLAIMED

CLAIMED (Component Library for AI, Machine Learning, ETL and Data Science) is a runtime and programming language agnostic Data & AI component framework.

DeepCausality

DeepCausality is a hyper-geometric computational causality library that enables fast and deterministic context-aware causal reasoning over complex multi-stage causality models.

DeepRec

DeepRec is a high-performance recommendation deep learning framework based on TensorFlow 1.15, Intel-TensorFlow and NVIDIA-TensorFlow.

DLRover

The mission of the DLRover Project is to make the distributed training of large AI models easy, stable, fast and green through the training of deep learning models on a distributed cluster.

Feathr

Feathr is an enterprise-grade, high-performance feature store.

FlagAI

FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale models.

Intersectional Fairness (ISF)

Intersectional Fairness (ISF) is a bias detection and mitigation technology for addressing intersectional bias, which is caused by the combinations of multiple protected attributes.

IREE

IREE (Intermediate Representation Execution Environment) is a sandbox-stage project of LF AI & Data Foundation.

LakeSoul

LakeSoul is a cloud-native Lakehouse framework developed by DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & batch processing.

Machine Learning eXchange

Machine Learning eXchange (MLX) is a Data and AI Assets Catalog and Execution Engine.

Monocle

AI Observability as a Service. A domain specific tracing + metrics framework for monitoring generative AI applications.

OAAX

OAAX or Open AI Accelerator Exchange, is an open standard designed to simplify the development and deployment of edge AI applications. It provides a unified framework for converting and optimizing machine learning models for execution on various hardware accelerators, or XPUs.

Open Model Initiative

The mission of OMI is to support open community development of openly licensed baseline AI models for image, video and audio generation that individuals and organizations can use and augment in their own solutions.

Open Platform for Enterprise AI

The mission of the Open Platform for Enterprise AI (OPEA) Project is to develop an ecosystem orchestration framework to efficiently integrate performant GenAI technologies and workflows leading to quicker GenAI adoption and business value.

OpenFL

OpenFL is a Python 3 library for federated learning that enables organizations to collaboratively train a model without sharing sensitive information.

Recommenders

Best practices on recommendation systems.

RosaeNLG

RosaeNLG is an open source project, template-based Natural Language Generation (NLG) automating the production of relatively repetitive texts based on structured input data and textual templates, run by a NLG engine.

RYOMA

The mission of the Project is to build a general LLM agentic framework designed for data analysis.

SapientML

SapientML Project is a Meta-Learning based AutoML initiative designed to enhance the success of the AI model creation process.

Unity Catalog

Unity Catalog is the industry's only universal catalog for data and AI.

1chipML

1chipML is an open source library for basic numerical crunching and machine learning for microcontrollers.

Acumos AI

Acumos AI is a platform and open source framework that makes it easy to build, share, and deploy AI apps.

BeyondML

BeyondML is a framework for developing sparse neural networks that can perform multiple tasks across multiple data domains.

DataPractices

DataPractices is a “Manifesto for Data Practices,” comprised of values and principles to illustrate the most effective, modern, and ethical approach to data teamwork.

Delta

DELTA is a deep learning based end-to-end natural language and speech processing platform.

Elastic Deep Learning (EDL)

EDL is an Elastic Deep Learning framework designed to help deep learning cloud service providers to build cluster cloud services using deep learning frameworks such as PaddlePaddle and TensorFlow.

OpenBytes

OpenBytes aims to facilitate wider sharing of, and collaboration with, data in the AI community through the promotion of data standards and formats and enabling contributions of data.

OpenDataology

OpenDataology is an open source dataset license compliance analysis project.

ShaderNN

ShaderNN is a lightweight deep learning inference framework optimized for Convolutional Neural Networks on mobile platforms.

TonY

TonY is a framework to natively run deep learning jobs on Apache Hadoop.

Xtreme1

Xtreme1 is the next generation open source platform for multi-sensory training data.