Sismo Hub Guide: Create Data Groups Programmatically

Developer tutorial

What’s inside?

This beginner-friendly tutorial will walk you through the creation of a .

Group Generators are functions that enable the creation of at the center of the Sismo protocol. Groups are the foundation of all that you can create with .

You can find all already existing Group Generators .

You can find the pull request associated with this tutorial . It will give you a good feeling of what we are going to create in this tutorial! 👌

Don't hesitate to join our and ping us during a hackathon. We would love to talk to you and meet you there. Don't hesitate to send your PR there as well, we will quickly review and merge it (usually under 1 hour).

What are Groups?

Groups are composed of Data Sources:

Web2 accounts: Twitter, GitHub, Telegram
Web3 accounts: Ethereum addresses, ENS, Lens handles

In order to create a Group, we need to build a . This tutorial will guide you through the process of creating one.

There are 3 different ways to define the data of a Group generator, you can use:

A hardcoded list of Data Source (e.g. )
(e.g. )
Already existing (e.g. )

If you want more info on Groups, check out this .

Group use-cases

Here is an instance of a Sismo Connect App that has been implemented using the proof-of-humanity Group:

This Sismo Connect App allows you to gate contents/features of your app to Proof of Humanity registrants without revealing the registered addresses.

Tutorial use-case

Here's a sample of the Group and the metadata you will get at the end of the tutorial:

Let's build our Group now 🧑‍💻

Setup of the local environment

First, clone the Sismo Hub repository and cd into it:

# using ssh
git clone git@github.com:sismo-core/sismo-hub.git
cd sismo-hub

You can now install all the dependencies with the yarn command:

# install dependencies
yarn

Creation of the Group Generator

Go to the group-generators/generators folder and create a new folder and a file inside.

# in a new terminal
# you are in sismo-hub

# cd into the folder
cd group-generators/generators
# create the group-generator folder for our new Group
mkdir tutorial-sismo-early-interactors
cd tutorial-sismo-early-interactors
# create the file
touch index.ts

You will build your Group in the index.ts file.

Before creating our tutorial use case, let's create a simple Group with a basic hardcoded list of accounts:

// group-generators/generators/tutorial-first-sismo-post-collectors/index.ts

import {
  ValueType,
  Tags,
  GroupWithData,
} from "topics/group";
import {
  GenerationContext,
  GenerationFrequency,
  GroupGenerator,
} from "topics/group-generator";

const generator: GroupGenerator = {
  generationFrequency: GenerationFrequency.Once,

  generate: async (context: GenerationContext): Promise<GroupWithData[]> => {

    const jsonListData0 = {
      "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045": "1",
      "0xF61CabBa1e6FC166A66bcA0fcaa83762EdB6D4Bd": "1",
      "0x2E21f5d32841cf8C7da805185A041400bF15f21A": "1",
    };

    return [
      {
        // give a name to your Group
        name: "tutorial-sismo-early-interactors",
        timestamp: context.timestamp,
        // add a small description explaining how to be eligible to your Group
        description: "Be an early interactor in Sismo community",
        // document the Group eligibility criterias more specifically
        specs: "Collect the first Sismo Lens post (https://lenster.xyz/posts/0x26e5-0x02) or vote to the first new Sismo DAO proposal (https://snapshot.org/#/sismo.eth/proposal/0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52)",
        // reference the final data we created
          // two different data formats in the Group
          // Lens account -> "dhadrien.lens": "1"
          // Ethereum account -> "0x95af97aBadA3b4ba443ff345437A5491eF332bC5": "1", 
        data: jsonListData0,
        valueType: ValueType.Info,
        tags: [Tags.User, Tags.Lens, Tags.Web3Social],
      },
    ];
  },
};

export default generator;

Both name of the folder and of the group has to be in kebab-case

Example: this-is-my-folder or this-is-my-group

Once your Group Generator is built, you will then need to reference it in the generators folder:

// group-generators/generators/index.ts

...
// you need to import your group generator
import tutorialSismoEarlyInteractors from "./tutorial-sismo-early-interactors"

import { GroupGeneratorsLibrary } from "topics/group-generator";

export const groupGenerators: GroupGeneratorsLibrary = {
  ...
  // you need to reference your group generator
  "tutorial-sismo-early-interactors": tutorialSismoEarlyInteractors,
};

You can now generate the first version of the tutorial-sismo-early-interactors Group!

# command to generate the Group
yarn generate-group tutorial-sismo-early-interactors

NB: You can add --help to know all the commands for generate-group

If the group generation went well, here is the log you should see on your terminal:

As you can see we successfully generated the Group, and it contains the 3 elements (red box), these are the 3 accounts you defined in the jsonListData0 variable.

Now that you have an overview of the process of creating a Group, let's move on to the tutorial use case! 🧑‍💻

// group-generators/generators/tutorial-first-sismo-post-collectors/index.ts

import { dataOperators } from "@group-generators/helpers/data-operators";
import { dataProviders } from "@group-generators/helpers/data-providers";
import {
  ValueType,
  Tags,
  FetchedData,
  GroupWithData,
} from "topics/group";
import {
  GenerationContext,
  GenerationFrequency,
  GroupGenerator,
} from "topics/group-generator";

const generator: GroupGenerator = {
  generationFrequency: GenerationFrequency.Once,

  generate: async (context: GenerationContext): Promise<GroupWithData[]> => {
    // 1. Instantiate the Lens provider
    const lensProvider = new dataProviders.LensProvider();
    // query all the collectors of the first Sismo Lens post
    // https://lenster.xyz/posts/0x26e5-0x02
    const collectors: FetchedData = await lensProvider.getWhoCollectedPublication({
      publicationId: "0x26e5-0x02",
    });

    // 2. Instantiate the Snapshot provider
    const snapshotProvider = new dataProviders.SnapshotProvider();
    // query all the voters of the Sismo space on snapshot
    // https://snapshot.org/#/sismo.eth/proposal/0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52
    const voters: FetchedData = await snapshotProvider.queryProposalVoters({
      proposal: "0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52"
    });

    // 3. Make a union of the two queried data
    const tutorialSismoActiveMembers = dataOperators.Union([
      collectors,
      voters,
    ]);

    return [
      {
        // give a name to your Group
        name: "tutorial-sismo-early-interactors",
        timestamp: context.timestamp,
        // add a small description explaining how to be eligible to your Group
        description: "Be an early interactor in Sismo community",
        // document the Group eligibility criterias more specifically
        specs: "Collect the first Sismo Lens post (https://lenster.xyz/posts/0x26e5-0x02) or vote to the first new Sismo DAO proposal (https://snapshot.org/#/sismo.eth/proposal/0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52)",
        // reference the final data we created
          // two different data formats in the Group
          // Lens account -> "dhadrien.lens": "1"
          // Ethereum account -> "0x95af97aBadA3b4ba443ff345437A5491eF332bC5": "1", 
        data: tutorialSismoActiveMembers,
        valueType: ValueType.Info,
        tags: [Tags.User, Tags.Lens, Tags.Web3Social],
      },
    ];
  },
};

export default generator;

And... 🥁 It's already finished!

As you may notice, the data field is where the Group of Data Sources is stored, and its type is a FetchedData:

export type FetchedData = {
    [address: string]: BigNumberish;
};

And here's an example of what a FetchedData object looks like:

{
  "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045": "1",
  "0xF61CabBa1e6FC166A66bcA0fcaa83762EdB6D4Bd": "1",
  "0x2E21f5d32841cf8C7da805185A041400bF15f21A": "1",
}

Here there is only "0x123..." type addresses but the fetchedData object can also contain other different types of accounts:

ENS (e.g. "sismo.eth":"1")
Lens handles (e.g. "sismo.lens":"1")
GitHub accounts (e.g. "github:leosayous21":"1")
Twitter accounts (e.g. "twitter:leosayous21":"1")
Telegram accounts (e.g. "telegram:pelealexandru":"1")

Last but not least, you may have noticed the "1" at the end of each account in a Group. This is called: the value and it allows you to add granularity to your Group. For our tutorial, we don't want to make a distinction between each account in the Group, but if we wanted to, we could set a value of 2 for all collectors for instance.

Here is an example of how to use this value: at Sismo, we use a Sismo Contributor ZK Badge for voting on Sismo DAO Proposals. There is 3 different value in the Group:

"1" --> get 1 voting power
"2" --> get 50 voting power
"3" --> get 500 voting power

Data Provider setup

# in a terminal
# you are in sismo-hub root

cp .example.env .env

Then you can add your own API key by adding a new line to this file:

# in the .env file
export <THE_DATA_PROVIDER_YOU_WANT_TO_USE>="<YOUR_API_KEY>"

And you need to export the variable in order for you to use it locally on your computer:

# in a terminal
# you are in sismo-hub root

source .env

You can now, once again, generate the tutorial-sismo-early-interactors Group:

# command to generate the Group
yarn generate-group tutorial-sismo-early-interactors

If the Group generation went well, here is the log you should see on your terminal:

The Group has been successfully generated and it contains 1550 accounts (red box) at the time of the tutorial. But you may have a different number depending on the number of new collectors from the Sismo Lens post.

As you can see on the screenshot, after the generation, the command returns us some information about the Group you just generated.

You can retrieve the Group data at the path indicated in yellow, and if you want to see a sample of the Group you just generated:

# in a terminal
# this will show you the first 10 line of the Group
head -10 <yellow-link>

Then the 2 others blue paths are API endpoints that allow you to get from the Group its data and metadata. To use the API:

Sismo Hub API setup

# in a new terminal
# at the Sismo Hub repository root
yarn api:watch

Finally, you should see what is displayed on the screenshot in Tutorial use-case

See your work integrated on Sismo

Your Group Generator is now ready, so it's time to push it in prod!

Then you will have to add a new remote to your repo:

git remote add fork <your_fork_repo_link.git>

You can also create a new branch for your work:

git checkout -b tutorial-sismo-early-interactor-group

In this tutorial you should have 2 different files changed:

1 file created: group-generators/generators/tutorial-sismo-early-interactors/index.ts
1 file changed: group-generators/generators/index.ts

Next, add all your changes and push them to your fork:

# git add the two files
git add .
git commit -m "feat: add tutorial-sismo-early-interactors group generator"
# if you created the branch tutorial-sismo-early-interactor-group
git push fork tutorial-sismo-early-interactor-group
# otherwise
git push

You will then see on your forked repository the following message, you can click on "Compare & pull request" to begin creating your pull request:

Finally, your Group will be available in the Sismo Hub and anyone will be able to use it to create Badges or Sismo Connect apps, great job! 💪

Use your Group

Now that you have your Group, you can build the following with it:

- - You can find here boilerplates to run on your computer.

Contribute to the Sismo Hub

PreviousFactory Guide: Create a Data Group in 5 Minutes NextSismo Hub Guide: Add a Data Provider to the Sismo Factory

Last updated 1 year ago