Comment on page
Sismo Hub Guide: Create Data Groups Programmatically
Developer tutorial
Group Generators are functions that enable the creation of Groups at the center of the Sismo protocol. Groups are the foundation of all that you can create with Sismo Connect Apps.
You can find the pull request associated with this tutorial here. It will give you a good feeling of what we are going to create in this tutorial! 👌
Don't hesitate to join our Dev Telegram and ping us during a hackathon. We would love to talk to you and meet you there. Don't hesitate to send your PR there as well, we will quickly review and merge it (usually under 1 hour).
Groups are composed of Data Sources:
- Web2 accounts: Twitter, GitHub, Telegram
- Web3 accounts: Ethereum addresses, ENS, Lens handles
In order to create a Group, we need to build a Group Generator. This tutorial will guide you through the process of creating one.
There are 3 different ways to define the data of a Group generator, you can use:
If you want to know how to contribute to the Sismo Hub by creating Groups don't hesitate to check the Contributor Guide.
Here is an instance of a Sismo Connect App that has been implemented using the proof-of-humanity Group:
Sismo Connect App example
This Sismo Connect App allows you to gate contents/features of your app to Proof of Humanity registrants without revealing the registered addresses.
After this tutorial, you will be able to create a Sismo Connect App from your Group through the Factory.
This tutorial will show you how to create a Group and make it usable for a Sismo Connect App through the Factory.
In the course of this tutorial, you will build your first Group named
tutorial-sismo-early-interactors, containing all the collectors of the first Sismo Lens post and all voters of one of the first Sismo Proposals.Here's a sample of the Group and the metadata you will get at the end of the tutorial:
Group data sample
Group Metadata
Let's build our Group now 🧑💻
First, clone the Sismo Hub repository and
cd into it:You can now install all the dependencies with the
yarn command:# install dependencies
yarn
Go to the
group-generators/generators folder and create a new folder and a file inside.# in a new terminal
# you are in sismo-hub
# cd into the folder
cd group-generators/generators
# create the group-generator folder for our new Group
mkdir tutorial-sismo-early-interactors
cd tutorial-sismo-early-interactors
# create the file
touch index.ts
You will build your Group in the
index.ts file.Before creating our tutorial use case, let's create a simple Group with a basic hardcoded list of accounts:
// group-generators/generators/tutorial-first-sismo-post-collectors/index.ts
import {
ValueType,
Tags,
GroupWithData,
} from "topics/group";
import {
GenerationContext,
GenerationFrequency,
GroupGenerator,
} from "topics/group-generator";
const generator: GroupGenerator = {
generationFrequency: GenerationFrequency.Once,
generate: async (context: GenerationContext): Promise<GroupWithData[]> => {
const jsonListData0 = {
"0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045": "1",
"0xF61CabBa1e6FC166A66bcA0fcaa83762EdB6D4Bd": "1",
"0x2E21f5d32841cf8C7da805185A041400bF15f21A": "1",
};
return [
{
// give a name to your Group
name: "tutorial-sismo-early-interactors",
timestamp: context.timestamp,
// add a small description explaining how to be eligible to your Group
description: "Be an early interactor in Sismo community",
// document the Group eligibility criterias more specifically
specs: "Collect the first Sismo Lens post (https://lenster.xyz/posts/0x26e5-0x02) or vote to the first new Sismo DAO proposal (https://snapshot.org/#/sismo.eth/proposal/0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52)",
// reference the final data we created
// two different data formats in the Group
// Lens account -> "dhadrien.lens": "1"
// Ethereum account -> "0x95af97aBadA3b4ba443ff345437A5491eF332bC5": "1",
data: jsonListData0,
valueType: ValueType.Info,
tags: [Tags.User, Tags.Lens, Tags.Web3Social],
},
];
},
};
export default generator;
Both name of the folder and of the group has to be in
kebab-caseExample:
this-is-my-folder or this-is-my-groupOnce your Group Generator is built, you will then need to reference it in the
generators folder:// group-generators/generators/index.ts
...
// you need to import your group generator
import tutorialSismoEarlyInteractors from "./tutorial-sismo-early-interactors"
import { GroupGeneratorsLibrary } from "topics/group-generator";
export const groupGenerators: GroupGeneratorsLibrary = {
...
// you need to reference your group generator
"tutorial-sismo-early-interactors": tutorialSismoEarlyInteractors,
};
You can now generate the first version of the
tutorial-sismo-early-interactors Group!# command to generate the Group
yarn generate-group tutorial-sismo-early-interactors
NB: You can add--helpto know all the commands forgenerate-group
If the group generation went well, here is the log you should see on your terminal:
Group generation succeeded
As you can see we successfully generated the Group, and it contains the 3 elements (red box), these are the 3 accounts you defined in the
jsonListData0 variable.Now that you have an overview of the process of creating a Group, let's move on to the tutorial use case! 🧑💻
As previously stated, we want to fetch all the collectors of the first Sismo Lens post and all voters of one of the first Sismo Proposals.
- The Lens Data Provider: it will allow you to fetch all the collectors of the first Sismo Lens Post. (1.)
If you want more info on how Data Provider works and how to create one, there is a tutorial for you here. This tutorial makes you create a Lens data provider and a function that fetch all Lens post collectors (It's a good time, isn't it 😉)
- The Snapshot Data Provider: it will allow you to fetch all the voters of one of the first Sismo proposals on Snapshot. (2.)
Once this data has been fetched, you'll need to merge it using Data Operators. A few different Data Operators that allow you to make operations on the data exist. Here you want to keep all the collectors and voters in the Group, so you will use the Union Data Operator. (3.)
// group-generators/generators/tutorial-first-sismo-post-collectors/index.ts
import { dataOperators } from "@group-generators/helpers/data-operators";
import { dataProviders } from "@group-generators/helpers/data-providers";
import {
ValueType,
Tags,
FetchedData,
GroupWithData,
} from "topics/group";
import {
GenerationContext,
GenerationFrequency,
GroupGenerator,
} from "topics/group-generator";
const generator: GroupGenerator = {
generationFrequency: GenerationFrequency.Once,
generate: async (context: GenerationContext): Promise<GroupWithData[]> => {
// 1. Instantiate the Lens provider
const lensProvider = new dataProviders.LensProvider();
// query all the collectors of the first Sismo Lens post
// https://lenster.xyz/posts/0x26e5-0x02
const collectors: FetchedData = await lensProvider.getWhoCollectedPublication({
publicationId: "0x26e5-0x02",
});
// 2. Instantiate the Snapshot provider
const snapshotProvider = new dataProviders.SnapshotProvider();
// query all the voters of the Sismo space on snapshot
// https://snapshot.org/#/sismo.eth/proposal/0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52
const voters: FetchedData = await snapshotProvider.queryProposalVoters({
proposal: "0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52"
});
// 3. Make a union of the two queried data
const tutorialSismoActiveMembers = dataOperators.Union([
collectors,
voters,
]);
return [
{
// give a name to your Group
name: "tutorial-sismo-early-interactors",
timestamp: context.timestamp,
// add a small description explaining how to be eligible to your Group
description: "Be an early interactor in Sismo community",
// document the Group eligibility criterias more specifically
specs: "Collect the first Sismo Lens post (https://lenster.xyz/posts/0x26e5-0x02) or vote to the first new Sismo DAO proposal (https://snapshot.org/#/sismo.eth/proposal/0xe280e236c5afa533fc28472dd0ce14e5c3514a843c0563552c962226cda05c52)",
// reference the final data we created
// two different data formats in the Group
// Lens account -> "dhadrien.lens": "1"
// Ethereum account -> "0x95af97aBadA3b4ba443ff345437A5491eF332bC5": "1",
data: tutorialSismoActiveMembers,
valueType: ValueType.Info,
tags: [Tags.User, Tags.Lens, Tags.Web3Social],
},
];
},
};
export default generator;
And... 🥁 It's already finished!
As you may notice, the
data field is where the Group of Data Sources is stored, and its type is a FetchedData:export type FetchedData = {
[address: string]: BigNumberish;
};
And here's an example of what a
FetchedData object looks like:{
"0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045": "1",
"0xF61CabBa1e6FC166A66bcA0fcaa83762EdB6D4Bd": "1",
"0x2E21f5d32841cf8C7da805185A041400bF15f21A": "1",
}
Here there is only "
0x123..." type addresses but the fetchedData object can also contain other different types of accounts:- ENS (e.g.
"sismo.eth":"1") - Lens handles (e.g.
"sismo.lens":"1") - GitHub accounts (e.g.
"github:leosayous21":"1") - Twitter accounts (e.g.
"twitter:leosayous21":"1") - Telegram accounts (e.g.
"telegram:pelealexandru":"1")
During the Group generation, all accounts are resolved into an Ethereum address through a resolver. There is a resolver for each type of Data Source.
Last but not least, you may have noticed the
"1" at the end of each account in a Group. This is called: the value and it allows you to add granularity to your Group. For our tutorial, we don't want to make a distinction between each account in the Group, but if we wanted to, we could set a value of 2 for all collectors for instance.Here is an example of how to use this value: at Sismo, we use a Sismo Contributor ZK Badge for voting on Sismo DAO Proposals. There is 3 different value in the Group:
- "1" --> get 1 voting power
- "2" --> get 50 voting power
- "3" --> get 500 voting power
For this tutorial, you only use Lens and Snapshot Data Providers which don't require API key. But for other Data Providers (e.g. BigQuery, Hive...) or resolvers (e.g. Twitter) an API key is required. So if you want to set an API key for the Data Provider you want to use, you need to create an environment variable. And here is how to do it:
# in a terminal
# you are in sismo-hub root
cp .example.env .env
Then you can add your own API key by adding a new line to this file:
# in the .env file
export <THE_DATA_PROVIDER_YOU_WANT_TO_USE>="<YOUR_API_KEY>"
And you need to export the variable in order for you to use it locally on your computer:
# in a terminal
# you are in sismo-hub root
source .env
For instance, If you want to add Twitter accounts to your group, you will need a Twitter API Key. Here's how to get one : https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api
You can now, once again, generate the
tutorial-sismo-early-interactors Group:# command to generate the Group
yarn generate-group tutorial-sismo-early-interactors
If the Group generation went well, here is the log you should see on your terminal:
Group generation succeeded
The Group has been successfully generated and it contains 1550 accounts (red box) at the time of the tutorial. But you may have a different number depending on the number of new collectors from the Sismo Lens post.
As you can see on the screenshot, after the generation, the command returns us some information about the Group you just generated.
You can retrieve the Group data at the path indicated in yellow, and if you want to see a sample of the Group you just generated:
# in a terminal
# this will show you the first 10 line of the Group
head -10 <yellow-link>
Then the 2 others blue paths are API endpoints that allow you to get from the Group its data and metadata. To use the API:
# in a new terminal
# at the Sismo Hub repository root
yarn api:watch
NB: You can go to http://127.0.0.1:8000/static/rapidoc/index.html to see the main endpoints of the Sismo Hub API
Your Group Generator is now ready, so it's time to push it in prod!
You don't have to store anything or launch anything on your side. You will just have to add a pull request to the Sismo Hub to see your Group Generator live! 😁
Fork Sismo Hub repository
Then you will have to add a new remote to your repo:
git remote add fork <your_fork_repo_link.git>
You can also create a new branch for your work:
git checkout -b tutorial-sismo-early-interactor-group
In this tutorial you should have 2 different files changed:
- 1 file created:
group-generators/generators/tutorial-sismo-early-interactors/index.ts - 1 file changed:
group-generators/generators/index.ts
Next, add all your changes and push them to your fork:
# git add the two files
git add .
git commit -m "feat: add tutorial-sismo-early-interactors group generator"
# if you created the branch tutorial-sismo-early-interactor-group
git push fork tutorial-sismo-early-interactor-group
# otherwise
git push
You will then see on your forked repository the following message, you can click on "Compare & pull request" to begin creating your pull request:
Click on "Compare & pull request" to begin creating your PR
You will finally see your branch being compared to the main branch of the Sismo Hub repository (blue boxes). You can review your changes and add a meaningful title and comments, and when you are happy with your PR, click "Create Pull request". 😇
Create your pull request
When your PR is merged your Group generator will be registered in the Sismo Hub, and your Group will be generated and sent onchain (through the Registry Tree).
Finally, your Group will be available in the Sismo Hub and anyone will be able to use it to create Badges or Sismo Connect apps, great job! 💪
Don't hesitate to join our Dev Telegram and ping us during hackaton. We would love to talk to you and meet you there. Don't hesitate to send your PR there as well, we will quickly review and merge it (usually under 1 hour).
Now that you have your Group, you can build the following with it:
You want to contribute but you don't have any ideas for Groups to create? Check out the current Sismo Hub GitHub's issues for some interesting ideas.
If you have any questions or you need help regarding your Group creation process, do not hesitate to join our Discord and ask us in #dev-support or ou Dev Telegram. We will be glad to answer you 🤗