Data Knobs | Experiment thru orthogonal knobs


Orthogonal Data Knobs

In many domains, companies have to run thousands of experiments to find plausible candidates. Data scientist team has to code experiment. But a team can only manage 5-10 or may be 50 experiments. Runnig hundreds of experiments and comparing models become unmanageble. Moreover the experiments are hidden behind data scientist desk. When they leave and new resource join, whole thing start over again.

Problems in which large number of experiments need to run, should be manage thru dials or knobs. Using knobs data scientist can use their statistics, domain knowledge and valida/invalidate hypothesis. The outcome of experiments are recorded and even if results are not fruitful, it increase knowledge base.

Knobs for experimentation

We can define experimentation problem as - we are given a pool of preprocessing methods, feature transformation, ML algorithms and hyper parameters. Our goal is to select the combination of these data processing and ML methods that produce the best model result for a given data set.

The system should deal with the messiness and complexity of data, automate feature selection, select machine learning (ML) algorithm to train a model. The system does it in such a manner that is efficient and robust and considers constraints not only about accuracy but memory, compute time, data need etc

As data pattern will continue to change and you want data scientist to make decision - features, model paramters, we define the solution in which data scientists can interact and explore in a semi-automated manner using orthogonal dials or knobs.

Orthogonal knobs are dials which data scientist or domain expert can tune. They can choose different features or normalize feature in diffeent manner, they can choose different algorithm or different loss function

They are similar to model hyper paramter, But model hyper paramters are only for model algorithms. Model hyper paramters are model algorithm code dependant

Philosphy behind data knobs are these are parameters that are bottoms up generated based on data and how these data is used in process. These are super set of hyper paramters as it let you choose features, featue transformation, data sources, loss functional computations etc

Problem can be mathmatically represented as:

Model(M)

Input
  • Dataset {Xi, Yi}
  • Objective function J(f) to evaluate model performance
  • Constraints: Data scientist time, accuracy, etc

    Output
  • A trained model in the form y = f(x)
  • We can describe this in form of y = f(x; α)
  • Where set α   =  [ α ₀, α ₁, α ₂, …, αₙ] are parameters of model
  • Processing

    Consider a vector θ. It includes all possible operations on data (e.g. ingestion, transformation, feature engineering, modeling, hyperparameter tuning)

    θ   =  [ θ ₁, θ ₂, …, θ ₙ]

    Note: For simplicity, we can consider all θ n as simple element operations. In elaborate settings, trees and graphs can be used to represent dependencies/hierarchy of operations.
    Refined Problem Statement

    We can define problem statement as - we have a pool of preprocessing methods, feature transformation methods, ML algorithms, and hyperparameters. The goal is to select the combination of knobs that produce the best results. Goal is to identify these knobs so that one can use different settings when data pattern changes.

    Goal
  • Efficiently find set of elements in θ that produce the best α
  • Enable building Orthogonal knobs O
  • Steps to implement
  • Intelligently and efficiently determine a set of values in θ that will produce results.
  • Automate execution of θ vector to produce α and evaluate the result
  • Enable creating higher-level θs and build dials O[] control
  • Once we define the θ vector, it simplify modeling and data science work. Now data scientist and domain expert focus on validating hypothesis, they are not worried to ensure whether some made short cut in feature transformation or made a mistake

    You get following benefits

  • Ability to run large number of experiment. Most experiment do not equire code changes. you change knobs settings.
  • Ability to run reproducuible experiments
  • Ability to log experiment outcome in meaningful manner - set of knobs and outcome. If someone has run experiment before in organization,they will know it. Team will build on each other experiments
  • Differential privacy blog


    Know about differential privacy at Differential privacy blog

    Learn about algorithms - K-Anonymizatio, T-Closeness, L-diversisty, Delta presence

    Learn about frameowrk to apply Differential privacy using data knobs

    From the blog

    Build Dataproducts

    How Dataknobs help in building data products

    Enterprises are most successful when they treat data like a product. It enable to use data in multiple use cases. However data product should be designed differently compared to software product.

    Be Data Centric and well governed

    Generative AI is one of approach to build data product

    Generative AI has enabled many transformative scenarios. We combine generative AI, AI, automation, web scraping, ingesting dataset to build new data products. We have expertise in generative AI, but for business benefit we define our goal to build data product in data centric manner. Our Product KREATE enable creation of data, user interface, AI assistant. Click to see it in action.

    Well Governed data

    Data Lineage and Extensibility

    To build a commercial data product, create a base data product. Then add extension to these data product by adding various types of transformation. However it lead to complexity as you have to manage Data Lineage. Use knobs for lineage and extensibility

    What is KREATE and KreatePro

    Kreate - Bring your Ideas to Life

    KREATE empowers you to create things - Dataset, Articles, Presentations, Proposals, Web design, Websites and AI Assistants Kreate is a platform inclide set of tools that ignite your creatviity and revolutionize the way you work. KReatePro is enterprise version.

    What is KONTROLS

    KONTROLS - apply creatvity with responsbility

    KONTROLS enable adding guardrails, lineage, audit trails and governance. KOntrols recogizes that different use cases for Gen AI and AI have varying levels of control requirements. Kontrols provide structure to select right controls.

    What is KNOBS

    KNOBS - Experimentation and Diagnostics

    Well defined tunable paramters for LLM API, LLM fine tuning , Vector DB. These parameters enable faster experimentation and diagosis for every state of GenAI development - chunking, embedding, upsert into vector DB, retrievel, generation and creating responses for AI Asistant.

    Kreate Articles

    Create Articles and Blogs

    Create articles for Blogs, Websites, Social Media posts. Write set of articles together such as chapters of book, or complete book by giving list of topics and Kreate will generate all articles.

    Kreate Slides

    Create Presentations, Proposals and Pages

    Design impactful presentation by giving prmpt. Convert your text and image content into presentations to win customers. Search in your knowledbe base of presentations and create presentations or different industry. Publish these presentation with one click. Generate SEO for public presentations to index and get traffic.

    Kreate Websites

    Agent to publish your website daily

    AI powered website generation engine. It empower user to refresh website daily. Kreate Website AI agent does work of reading conent, website builder, SEO, create light weight images, create meta data, publish website, submit to search engine, generate sitemap and test websites.

    Kreate AI Assistants

    Build AI Assistant in low code/no code

    Set up AI Assistant that give personized responss to your customers in minutes. Add RAG to AI assistant with minimal code- implement vector DB, create chunks to get contextual answer from your knowlebase. Build quality dataset with us for fine tuning and training a cusom LLM.

    Create AI Agent

    Build AI Agents - 5 types

    AI agent independently chooses the best actions it needs to perform to achieve their goals. AI agents make rational decisions based on their perceptions and data to produce optimal performance and results. Here are features of AI Agent, Types and Design patterns

    Develop data products with KREATE and AB Experiment

    Develop data products and check user response thru experiment

    As per HBR Data product require validation of both 1. whether algorithm work 2. whether user like it. Builders of data product need to balance between investing in data-building and experimenting. Our product KREATE focus on building dataset and apps , ABExperiment focus on ab testing. Both are designed to meet data product development lifecycle

    Innovate with experiments

    Experiment faster and cheaper with knobs

    In complex problems you have to run hundreds of experiments. Plurality of method require in machine learning is extremely high. With Dataknobs approach, you can experiment thru knobs.

    Why knobs matter

    Knobs are levers using which you manage output

    See Drivetrain appproach for building data product, AI product. It has 4 steps and levers are key to success. Knobs are abstract mechanism on input that you can control.

    Our Products

    KreateBots

  • Pre built front end that you can configure
  • Pre built Admin App to manage chatbot
  • Prompt management UI
  • Personalization app
  • Built in chat history
  • Feedback Loop
  • Available on - GCP,Azure,AWS.
  • Add RAG with using few lines of Code.
  • Add FAQ generation to chatbot
  • KreateWebsites

  • AI powered websites to domainte search
  • Premium Hosting - Azure, GCP,AWS
  • AI web designer
  • Agent to generate website
  • SEO powered by LLM
  • Content management system for GenAI
  • Buy as Saas Application or managed services
  • Available on Azure Marketplace too.
  • Kreate CMS

  • CMS for GenAI
  • Lineage for GenAI and Human created content
  • Track GenAI and Human Edited content
  • Trace pages that use content
  • Ability to delete GenAI content
  • Generate Slides

  • Give prompt to generate slides
  • Convert slides into webpages
  • Add SEO to slides webpages
  • Content Compass

  • Generate articles
  • Generate images
  • Generate related articles and images
  • Get suggestion what to write next