Sismo Docs
  • Welcome to Sismo
    • What is Sismo?
  • Build with Sismo Connect
    • Overview
    • Installation
    • Sismo Connect Cheatsheet
    • Tutorials
      • Get Your appId - Create a Sismo Connect App
      • Onchain Tutorial (1/2): Code Your Airdrop Contract With Privately-Aggregated Data
      • Onchain Tutorial (2/2): Deploy Your Airdrop Contract
    • Technical Documentation
      • Sismo Connect Configuration
      • Auths
      • Claims
      • Signature
      • Packages
        • Sismo Connect Client: Request
        • Sismo Connect React: Request
        • Sismo Connect Server: Verify Offchain
        • Sismo Connect Solidity Library: Verify Onchain
    • FAQ
  • Data Groups
    • Overview
    • Tutorials
      • Factory Guide: Create a Data Group in 5 Minutes
      • Sismo Hub Guide: Create Data Groups Programmatically
      • Sismo Hub Guide: Add a Data Provider to the Sismo Factory
    • Sismo Hub
      • Sismo Hub Repository
        • Group Generators
        • Data Providers
        • Data Operators
        • Command Line Interface
      • Accounts Registry Tree
  • Data Vault
    • Overview
    • Vault Identifiers
    • Proving Schemes
      • Hydra-S1
      • Hydra-S2
    • Commitment Mapper
  • Resources
    • Deployed Contract Addresses
    • Sismo API
      • API Links
      • Query From a Client
      • Group
        • Get groups
        • Get group snapshots
      • Common Parameters
      • Advanced Filtering
      • Transaction
  • Links
    • Sismo Landing Page
    • Sismo Factory
    • Sismo App Store
    • Sismo Builder Resources
    • GitHub
    • Discord
    • Twitter
    • Blog
Powered by GitBook
On this page
  • What's inside?
  • What are Data Providers?
  • Tutorial use case
  • Tutorial
  • Setup of the local environment
  • Creation of the Data Provider
  • Make your Data Provider usable by anyone on the Factory
  • See your work integrated on Sismo
  • Contribute to the Sismo Hub
  1. Data Groups
  2. Tutorials

Sismo Hub Guide: Add a Data Provider to the Sismo Factory

Developer tutorial

PreviousSismo Hub Guide: Create Data Groups ProgrammaticallyNextSismo Hub

Last updated 1 year ago

What's inside?

This tutorial will explain how to create a in the . Users (through the ) and developers (through the ) will be able to use your Data Provider in order to create .

Group Generators are functions that enable the creation of groups at the center of the Sismo protocol. As groups need to contain data, we need Data Providers to fetch data from external APIs (e.g, Lens, GitHub, Snapshot).

You can find all already existing Data Providers .

By following this tutorial you will learn how to create a new Data Provider to fetch the data for your desired group.

Data providers are at the foundation of the Sismo Hub architecture. They are designed to abstract the complexity of fetching external APIs for future devs or group creators gathering data. They are meant to be easy to use and reliable for future group generation.

You can find the pull request associated with this tutorial .

What are ?

Data Providers enable other group creators to reuse specific data fetching logic, such as calls to APIs.

are composed of Data Source:

  • Web2 accounts: Twitter, GitHub, Telegram

  • Web3 accounts: Ethereum addresses, ENS, Lens handle

When creating a group, you need to create a . There are 3 different ways to do so:

  • Enter a hardcoded list of accounts

  • Use an already existing

  • Use a to fetch accounts

Moreover, Data Providers are useful because they enable automatic and frequent group updates, whereas if the group was made of a hardcoded list of accounts it wouldn't be automatic.

For example:

const snapshotProviderData0 = await snapshotProvider.queryProposalVoters({
      proposal: "0x864685d094312bc51eeb01af69e83dc7c496108cca54a15b491d79e9ce389889"
    });

This will fetch all the voters of the CoW Swap proposal number 23 (CIP-23)

Tutorial use case

This tutorial will show you how to create a Data Provider and make it usable in the Factory.

We will take the example of a Lens Data Provider on top of the Lens API and build our first query that gets the list of collectors of a specific Lens post.

Tutorial

Setup of the local environment

First, you need to clone the Sismo Hub repository

# using ssh
git clone git@github.com:sismo-core/sismo-hub.git
cd sismo-hub

# install dependencies
yarn

Creation of the Data Provider

Let's build our Data Provider now 🧑‍💻

Go to the group-generators/helpers/data-providers folder and create a new folder:

# in a new terminal
# you are in sismo-hub

cd ./group-generators/helpers/data-providers
# create the data-provider folder
mkdir tutorial-lens
cd tutorial-lens

For our case in this tutorial, we will split our Data Provider into 3 different files:

  • index.ts: [mandatory] Contains all the functions you make available to use in a group generator. It will help format data that come from your API queries.

  • types.ts: [mandatory] The file where you can define all the types you need.

  • queries.ts: Define all your request to the API you want to use.

However, keep in mind that it's totally up to you to choose the structure you want for the files of your data provider. For this case, we found that it was better to split the Data Provider into these 3 files, but feel free to structure it however you like.

Only index.ts and types.ts are mandatory.

Let's start coding! 🧑‍💻

First, you need to define which query and which types you will use. From what we see in the Lens API docs, the whoCollectedPublication() query use and return some types, so you need to implement them in your data provider.

First, you can create a query that will use this type in the query.ts file:

# create the queries file
touch queries.ts
// group-generators/helpers/data-providers/tutorial-lens/queries.ts

// import gql in order to have prettier written requests
import { gql } from "graphql-request";
// import your previously created types
import { GetWhoCollectedPublicationType } from "./types";
// import the GraphQLProvider class
import { GraphQLProvider } from "@group-generators/helpers/data-providers/graphql";

// function that contains the query
export const getWhoCollectedPublicationQuery = async (
  graphqlProvider: GraphQLProvider,
  publicationId: string,
  cursor: string
): Promise<GetWhoCollectedPublicationType> => {
  // query
  return graphqlProvider.query<GetWhoCollectedPublicationType>(
    gql`
      query whoCollectedPublication($request: WhoCollectedPublicationRequest!) {
        whoCollectedPublication(request: $request) {
          items {
            address
          }
          pageInfo {
            prev
            next
            totalCount
          }
        }
      }
    `,
    {
      // the request will fetch 50 data by 50
      request: {
        // the id of the Lens post
        publicationId: publicationId,
        // number of data to fetch
        limit: 50,
        // this parameter allows specifying from which index you decide to fetch the data
        // example: cursor 50 & limit 100 => you will fetch the [50;100] data
        ...(cursor ? { cursor } : {}),
      },
    }
  );
};

Once done, create the types.ts file:

# create the types definitions file
touch types.ts

Then define all the types you need when using the whoCollectedPublication() query:

// group-generators/helpers/data-providers/tutorial-lens/types.ts

// this type informs us which parts of the set of elements we have fetched.
// it allows us to do pagination
export type PageInfo = {
  prev: string;
  next: string;
  totalCount: number;
};

export type Wallet = {
  address: string;
};

export type GetWhoCollectedPublicationType = {
  whoCollectedPublication: {
    items: Wallet[];
    pageInfo: PageInfo;
  };
};

// this type will be used in the index.ts file following the tutorial
export type PublicationId = {
  publicationId: string;
};

Because Lens API is a GraphQL API, you can use the GraphQL Data Provider already created in the Hub in order to create the query.

As you can see, this query allows fetching 50 users at a time. So you need to create a function that will iterate on this query in order to fetch all the users.

All of this occurs in the index.ts file:

# create the index file
touch index.ts
// group-generators/helpers/data-providers/tutorial-lens/index.ts

// import the queries
import {
  getWhoCollectedPublicationQuery,
} from "./queries";
// import the types
import {
  GetWhoCollectedPublicationType,
  PublicationId,
  Wallet
} from "./types";
// import the GraphQL Provider class
import { GraphQLProvider } from "@group-generators/helpers/data-providers/graphql";
// Import the FetchedData type
import { FetchedData } from "topics/group";

export class TutorialLensProvider extends GraphQLProvider {
  constructor() {
    super({
      // url of the API
      url: "https://api.lens.dev",
    });
  }
  
  // method that create the fetchedData object
  public async getWhoCollectedPublication(
    publication: PublicationId
  ): Promise<FetchedData> {
    const dataProfiles: FetchedData = {};
    for await (const item of this._getWhoCollectedPublication(publication)) {
      dataProfiles[item.address] = 1;
    }
    return dataProfiles;
  }

  // method that iterate on the getWhoCollectedPublicationQuery
  private async *_getWhoCollectedPublication({
    publicationId,
  }: PublicationId): AsyncGenerator<Wallet, void, undefined> {
    let cursor = "";
    let lensCollectors: GetWhoCollectedPublicationType;
    // fetch data as long as there is data
    do {
      lensCollectors = await getWhoCollectedPublicationQuery(
        this,
        publicationId,
        cursor
      );
      yield* lensCollectors.whoCollectedPublication.items;
      cursor = lensCollectors.whoCollectedPublication.pageInfo.next;
    } while (lensCollectors.whoCollectedPublication.items.length > 0);
  }
}

It is very important to pass an object that contains all your argument as argument of the function that the Factory will use.

Indeed, when the Factory create the group using your data provider, it will use take all the arguments gave by the user as input and create an object from it. Then it will call your function with this object, like this:

const lensProviderData0 = await lensProvider.getWhoCollectedPublication({
    publicationId: "0x26e5-0x09"
});
// for this case there is only 1 argument

That is why the argument of the function used by the Factory must be an object that contains all the possible arguments of your function.

For example in our case we needed to define this type:

export type PublicationId = {
  publicationId: string;
  // you can add other arguments here
};

Indeed, a Group Generator returns a group that is composed of metadata:

{
  name: "test",
  timestamp: context.timestamp,
  description: "This is a test group",
  specs: "",
  data: dataProfiles,
  valueType: ValueType.Score,
  tags: [],
}

The data field is where the group of accounts is stored, and its type is a FetchedData.

export type FetchedData = {
    [address: string]: BigNumberish;
};

And here's an example of how a FetchedData object looks like:

{
  "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045": "1",
  "0xF61CabBa1e6FC166A66bcA0fcaa83762EdB6D4Bd": "1",
  "0x2E21f5d32841cf8C7da805185A041400bF15f21A": "1",
}

Here there is only "0x123..." type addresses but the fetchedData object can contains also:

  • ENS: "vitalik.eth"

  • Lens handle: "stani.lens"

  • Twitter account: "twitter:Sismo_eth"

  • GitHub account: "github:leosayous21"

  • Telegram account: "telegram:pelealexandru"

Finally, you need to reference your newly created Data Provider in the data-provider folder in order to use it anywhere in the Sismo Hub:

// group-generators/helpers/data-providers/index.ts

...
import { TutorialLensProvider } from "./tutorial-lens";

export const dataProviders = {
  ...
  TutorialLensProvider,
};

Congrats 🎉 You've just created your first Data Provider!

But now that you have finished, you probably want to test it right?

Make your Data Provider usable by anyone on the Factory

To do so you need to create an interface-schema.json file that contains all the information the Factory needs to use your function and display relevant information.

# create the interface-schema file
touch interface-schema.json
// group-generators/helpers/data-providers/tutorial-lens/interface-schema.json

{
  "name": "Tutorial Lens",
  "iconUrl": "",
  "providerClassName": "TutorialLensProvider",
  "functions": [
    {
      "name": "Get collectors of",
      "functionName": "getWhoCollectedPublication",
      "countFunctionName": "getPublicationCollectorsCount",
      "description": "Returns all collectors of a specific publication",
      "args": [
        {
          "name": "publication identifier",
          "argName": "publicationId",
          "type": "string",
          "example": "0x26e5-0x03",
          "description": "A Lens publication identifier in the following format {profileId}-{publicationId}. https://docs.lens.xyz/docs/get-publication"
        }
      ]
    }
  ]
}

Then you have to reference your schema:

// group-generators/helpers/data-providers/index.ts

...
import tutorialLensProviderSchema from "./tutorial-lens/interface-schema.json";

export const dataProvidersInterfacesSchemas = [
  ...
  tutorialLensProviderSchema,
];

You're almost finished! 😉

As you may have noticed, in the interface-schema.json there is a countFunctionName. This is called a count function. Count functions allow the Factory to know how many accounts a request will fetch and thus to know if the data that the user wants to fetch exists.

Let's create it:

// group-generators/helpers/data-providers/tutorial-lens/index.ts

...  
  public async getPublicationCollectorsCount(
    publication: PublicationId
  ): Promise<number> {
    const publicationStats = await getPublicationStatsQuery(
      this,
      publication.publicationId
    );
    return publicationStats.publication.stats.totalAmountOfCollects;
  }
// group-generators/helpers/data-providers/tutorial-lens/queries.ts

...
export const getPublicationStatsQuery = async (
  graphqlProvider: GraphQLProvider,
  publicationId: string
): Promise<GetPublicationStatsType> => {
  return graphqlProvider.query<GetPublicationStatsType>(
    gql`
      query PublicationStats($request: PublicationQueryRequest!) {
        publication(request: $request) {
          ... on Post {
            stats {
              totalAmountOfCollects
            }
          }
          ... on Comment {
            stats {
              totalAmountOfCollects
            }
          }
          ... on Mirror {
            stats {
              totalAmountOfCollects
            }
          }
        }
      }
    `,
    {
      request: {
        publicationId: publicationId,
      },
    }
  );
};
// group-generators/helpers/data-providers/tutorial-lens/types.ts

...
export type GetWhoMirroredPublicationType = {
  profiles: {
    items: ProfileType[];
    pageInfo: PageInfo;
  };
};

This count function (getPublicationCollectorsCount) will return the number of collectors for a specific Lens post. It allows the Factory to verify if the publicationId given by the user is correct but also to display the number of accounts the query will fetch on the website.

Finally, you have to reference your count function in the dataProvidersAPIEndpoints:

// group-generators/helpers/data-providers/index.ts

export const dataProvidersAPIEndpoints = {
  ...
  // add your data provider to the /data-provider-interfaces end of the Sismo Hub API
  TutorialLensProvider: {
    // add your count function to your data provider on the Sismo Hub API
    getPublicationCollectorsCount: async (_: any) =>
      new TutorialLensProvider().getPublicationCollectorsCount(_),
  },
};

By adding this interface file and the count function, you just allowed the factory to:

  • Fetch information about your newly created Data Provider. It can be now displayed (red box) and used for Factory users when adding eligible accounts.

  • Call the right count function (regarding the user selection).

If you wish to choose the collectors for a Lens post, you would need to input the post's ID. After clicking the 'Add' button, the Factory will invoke your count function to verify the post's validity and display the number of eligible accounts at the bottom of the screen (see the example above).

You have finally set up your Data Provider for the Factory, congrats! 🎉

For our case, we use the Lens API which doesn't require any access token. However, for many APIs, that's not the case, so if you want to set an access token for the API you want to use, you need to create an environment variable. Here is how to do it:

# you are in sismo-hub root
cp .example.env .env

Then you can add your own access token by adding a new line to this file:

# in the .env file
export YOUR_DATA_PROVIDER_API_KEY="<API_KEY>"

And you need to export the variable in order for you to use it locally on your computer:

source .env

You are now able to use it on the Sismo Hub, here is the syntax:

process.env.YOUR_DATA_PROVIDER_API_KEY

See your work integrated on Sismo

Your Data Provider is now ready, so it's time to push it into prod!

Then you will have to add a new remote to your repo:

git remote add fork <your_fork_repo_link.git>

You can also create a new branch for your work:

git checkout -b tutorial-lens-data-provider

In this tutorial you should have 5 different files changed:

  • 4 files created:

    • group-generators/helpers/data-providers/tutorial-lens/index.ts

    • group-generators/helpers/data-providers/tutorial-lens/queries.ts

    • group-generators/helpers/data-providers/tutorial-lens/types.ts

    • group-generators/helpers/data-providers/tutorial-lens/interface-schema.json

  • 1 file changed

    • group-generators/helpers/data-providers/index.ts

Next, add all your changes and push them to your fork:

# git add all your changes except the group you made to test your data provider
git add <all-your-files-except-the-test-group>
git commit -m "feat: add tutorial-lens data provider"
# if you created the branch tutorial-lens-data-provider
git push fork tutorial-lens-data-provider
# otherwise
git push

Don't forget to only add the files related to the Data Provider, and do not add the group you made for testing your Data Provider.

You will then see on your forked repository the following message, you can click on "Compare & pull request" to begin creating your pull request:

When your PR is merged, you'll be able to see your Data Provider implemented in the Sismo Factory

Finally, your Data Provider will be available in the Sismo Hub and anyone will be able to use it to create groups. Great job! 💪

Contribute to the Sismo Hub

If you want to create a that contains all the voters of your last DAO proposal on Snapshot, simply use the queryProposalVoters function from the in your by giving the Proposal Identifier as an argument.

If you want to know more on Data Providers, check out this . Alternatively, you can check out the before creating one yourself.

At the end of this tutorial, it will be up to you to do a pull request on the to see your Data Provider integrated and be usable in the (green box) and in the Sismo Hub (red box) 🙌

You can install yarn .

It is worth noting that the final variable you return at the end of your getWhoCollectedPublication function is of type FetchedData. This type is needed if you want to create .

That's why must be the format of the variable you return.

To do this you will have to create a group (only for testing) and import your Data Provider into it, so you can try your request. If you don't know how to create a group, check out the group tutorial .

Okay, so now that your data provider is finished and tested, it can be used by all devs in the Sismo Hub. The next step is making it available to anyone in the . Pretty exciting, isn't it? 🤩

If you want more info on this schema, check this on the Data Provider page.

NB: To be able to use the API key in the Sismo Hub when your badge is deployed, please contact us in #dev-support () after creating your PR so we can add your access token to our infrastructure.

You don't have to store or launch anything on your side. You will just have to add to see your Data Provider live! 😁

First, you will have to the :

You will finally see your branch being compared to the main branch of the repository (blue boxes). You can review your changes and add a meaningful title and comments, and when you are happy with your PR, click "Create Pull request". 😇

For example, here is a valid pull request: (The PR is closed because it's only used as an example for this tutorial)

If your Data Provider requires an API key, contact the team on the in #dev-support in order for us to make your API key usable in the main Sismo Hub and on the Factory.

You will also see it in the Sismo Hub API:

You want to contribute, but you don't have any ideas for Data Providers to create? Check out the current , as you will find some interesting ideas for Data Providers to implement.

If you have any questions or you need help regarding your Data Providers creation process, do not hesitate to join our and ask us in #dev-support or ou . We will be glad to answer you 🤗

Data Provider
Sismo Hub
Factory
Sismo Hub
Group Generators
here
here
Data Providers
Groups
Group Generator
Group Generator
Data Provider
group
Snapshot Data Provider
Group Generator
page
Contributor Guide
Sismo Hub
Factory
here
Group Generators
FetchedData
here
Factory
Sismo Discord
a pull request to the Sismo Hub
fork
Sismo Hub repository
Sismo Hub
https://github.com/sismo-core/sismo-hub/pull/1407
Sismo Discord
https://hub.sismo.io/data-provider-interfaces
Sismo Hub GitHub's issues
Discord
Dev Telegram
Use of the data provider
Data Provider selection
Run the count function
Fork Sismo Hub repository
Click on "Compare & pull request" to begin creating your PR
Create your pull request
section