Goals
Farmstack Managed Connector that will be used by multiple users for hosting their own connector on DG server acrchitecture
Managed connector by Digital Green to enable users to experience data sharing with usage control restricting usage of data by a simple application
Chain of thoughts during discussion
Self-hosted participant software installer → generates UI → goes to server → fires up connectors → run connectors → data exchange through CSV files
One server (EC2 instance) used for multiple connectors hosted for multiple/single users
Test the capacity of instance and configuration for a limit on connectors that could be hosted
Experiment on possibilities and limitations of this idea
Take 4 GB of RAM and observe number of connectors
Technologies to be used:
Docker containers with or without Kubernetes
[ OR ]
Kubernetes only
Limit of connectors per user
Have to decide on a number?
Server (EC2 instance) will be destroyed after all connectors running inside are stopped
Should be automated
Multi-threaded creation of connectors
How should it be managed?
Technology?
System Design?
Database can be anywhere
Need more info?
Timeline of each connector
Destroy the connector after it’s time is over
Expanding capacity using Lambda
Need more info?
Celery worker to observe/ manage visualization
Finalize on class of machines for AWS instance
Working Pieces Readily Available
setup.py that does all of the above (however, have to experiment for scalablity) (link)
Loom video by sagar (link)
Video by Waseem (link)
✅ Action Items
Action | Description | Owner | Due date | Jira ticket | |
|---|---|---|---|---|---|
| 1 |
| ||||
| 2 |
| ||||
| 3 |
|
Add Comment