Palmtree
All case studies

Synthetic Data Generator

To generate a large volume of synthetic data which can be used for training vertical specific machine learning models.

  1. Started fromOperational databases
  2. Palmtree builtAI, LLMs and agents
  3. ResultSynthetic training data for healthcare, ad-tech and Salesforce types

Problem

To generate a large volume of synthetic data which can be used for training vertical specific machine learning models

Challenges

  • Generate large volumes of data in a distributed environment
  • Support data generation in different formats
  • Generate vertical specific data. For example, Healthcare, Advertising verticals

Solution

  • JSON schema for synthetic data generation is created using Datagen user interface
  • Datagen Service coordinates synthetic data generation
  • Synthetic Healthcare data is generated using Healthcare service
  • Synthetic Pixalate data is generated using Pixalate service
  • Synthetic Mobile app-ads.txt data is generated using AppAdsText service
  • Data generation also happens for Salesforce datatypes
  • All datagen services are dockerized and run in a Kubernetes cluster environment
  • Generated data are stored in Google Cloud Storage

How it fits together

Inside Kubernetes, the Rails Datagen UI stores jobs in PostgreSQL and calls the Spring Datagen service, which fans out to healthcare, Pixalate and other Docker services writing to a shared persistent disk. A storage writer and an on-demand Dataproc cluster with a field generator write generated data to Cloud Storage, which the UI serves back.
Select the diagram to open it full size.