Farmstack Tech Master Doc

Modules:

FS Core
FS Central
FS Participant

Points to work on:

Architecture of FS and tools required
1. Service oriented architecture vs Monolithic
2. Entities in FS - attributes, functions, operations
Database structure of FS entities (about connectors, policies, logs, users etc)
Wrappers around the entities available from Fraunhofer

Architecture of FS (other than connectors):

Decision points:
1. Service oriented vs monolith vs something else
  1. Is distributed DB must?
  2. Trade off: testability/ deployability vs faster development time
  3. Feasibility: Distributed system
  4. Collaboration: with dev partners like Fraunhofer and other potential partners
  5. Possibility to refactor entire codebase later
Tools
1. Deployment scripts/commands/plugins
2. Data provider and consumer custom applications (ex: NodeJS, Python, Java).
3. User management frontend and backend.
4. User console for:
  1. Deploying instances
  2. Interacting with deployment servers.
  3. Log for the user for services.

Important entities of FS:

Connectors:
1. Fraunhofer maintains the information model (declarative as well as programmatic) of the connectors/ usage policies/ contracts etc which defines the fields and attributes
2. DG should maintain:
  1. Functionality to extract the attributes and create entries in DB that can be referenced later
  2. Configurable templates of the connectors
  3. Wrapper to easily instantiate connector(s) from the templates
  4. Functionality to orchestrate through a frontend UI
3. Important categories of attributes:
  1. Type (provider vs consumer)
  2. Status (active, inactive, paused)
  3. Certificate
  4. Owner (org, person, certificate etc)
  5. Structure (container #s, sequence etc)
  6. Usage policies (type, metadata, status etc)

Certificates (in the section below CA refers to certificate authority and CnA will be Certification Authority):
1. Open questions:
  1. Certification Vs Certificate Authority - what Fraunhofer does maintain is certification authority (authority on process of creating certificates rather than creating certificates)
    1. We don’t need certification from IDSA to operate a connector
  2. Will DG + Fraunhofer be a CA or CnA?
    1. No, DG won’t be CA. The participant organisation doesn’t need a certificate since the access is given after manual verification
  3. What are the services to obtain certificates for being a CA? Or is it a manual process?
    1. Question is redundant as DG is not becoming a CA. FS needs to provide way to install TLS certificates to the connectors which can be obtained from ACME2 and a CA that needs to be decided (paid service)
  4. Do the certificates need to be part of DB maintained by FS central/ core?
    1. No, don’t think FS needs to have DB of certificates
    2. ??FS might need a DB of metadata of certificates??
Policies:
1. Fraunhofer maintains the information model that is serialized in json-ld
2. DG maintains:
  1. Define the workflow -> web sequence diagram for FS
  2. Functionality to extract the attributes and write to DB that can be referenced later
  3. Configurable templates of the policies
  4. ??Wrapper to easily instantiate static policies ??
  5. ??Wrapper for dynamic policies - with Fraunhofer??
  6. Functionality to orchestrate through a frontend UI
3. Important Attributes:
  1. Type
  2. Status
  3. Certificate
  4. Owner
  5. Structure (can be multiple contracts)
  6. Association with connector
Users
Organisations
Data resource (metadata of data)
Package (node/python app + docker container)
1. Is there a need to prove ownership of a package by the respective connector/ org?
2. What about the scenarios where the users insist on co-owning (multiple signatures)?
Participant host server
Central host server

Tools/ services:

Logging service:
1. Functionality

User	Requirements	Log viewer	Logs creation
Participant user	Wants to monitor status of the connectors etc	1) Status (active, paused) 2) Errors 3) Warnings 4) Notification ….. N) Data volume	1) Connector Flushed out if the transaction is successful Parsed, extracted, transformed , archived for future reference
Participant user	Wants to debug if connector has stalled
Central	Wants to monitor network level activity	1)

Participant logs:
1. Low level logs (to be flushed out):
  1. Connector logs: Lowest level logs generated to log files through the connector (debug logs)
  2. Package logs: Logs generated by the app while providing/ consuming data to the connector (if any running on the provider)
2. Participant web console logs: High level logs to be provided to the admin/root of the web admin console (status, data volume,

User Management:

To be stored according to the framework/application we choose, ex OpenLDAP, ApacheDS, OpenDJ, etc.

User journey:

User signup and onboarding. (Backend for central storage of Users)
Pre Provisioning the Deployment Server - Get required server information (SSH key, Cloud Service Provider API Key (Currently for AWS)) from Participant. (We might need some server management tool installed in case of personal/datacenter server)
Provision the deployment machines using some kind of middleware (AWS CLI, Terraform or other third party interfacing tool), install prerequisites and configure settings for deployment.
Containerize the application (R&D required)
Generate certificates, configurations - policies, uc, authentication
Configure required yml and xml files. (including data sources)
Clone the fs-core (connectors) on the provisioned server(s).
Attach the generated configurations to the cloned fs-core.
Run the docker machine.

Section 1 - User Signup and Onboarding: