This is a collection of skeleton projects and code generators used to get a new
user started with Zendro. To get you started minimal configuration is needed.
Be aware that this StarterPack provides a minimal working example using a single postgres database. To run Zendro in a real production environment and support other storageTypes more configuration is needed.
You should have basic knowledge of the following technology-stack:
Note, that this project is meant to be used on a *nix system, preferably
Linux.
First you need to git clone this project into a local directory on your host
system:
git clone https://github.com/Zendro-dev/ZendroStarterPack.git
The skeleton GraphQL server and the skeleton single page application server projects are managed as different git repositories. "Skeleton" means that these projects provide all the code needed to start a server, but actually have no code particular to any data model.
To download the skeleton projects as well as graphiql-auth, a standalone GraphiQL deployment, you can run
yarn setup
# -g: graphql-server branch | default: master
# -s: single-page-app branch | default: main
# -i: graphiql-auth branch | default: master
This will add the desired versions of the skeleton projects to the directory. You should now have three folders graphql-server, single-page-app and graphiql-auth in your StarterPack root directory.
To avoid having to install the Zendro graphql-server code-generator on your host system we provide a dedicated Docker image in which two code generators are installed and ready to be used. Run
yarn codegen:build
Within the directory ./data_model_definitions you can place your data model
definitions in respective JSON files. To learn more about how to define data
models with Zendro please see our manual and
documentation.
We strongly recommend using the default data models user.json and
role.json. This gives you a ready to login and secure set of servers.
If you choose to follow the recommendation, you should edit the Sequelize
seeder ./seeders/20190225162439-create_roles_n_users.js to create your
default admin-user and default roles.
If modified it is most important that you then copy the seeder into the graphql-server code dir:
cp -r ./seeders ./graphql-server
Using the dedicated Docker image in which the code generators are installed you
can invoke them on the data model definitions you placed in the
data_model_definitions directory.
yarn codegen:run
graphql-server-model-codegen -m -f /opt/data_model_definitions -o /opt/graphql-server
exit
Whenever you make changes to your data model definitions you should rerun the above command.
The SPA will automatically read from your data_model_definitions folder and generate the needed code. See SPA README for more information.
Be very carefull when running the code generators multiple times on the same data model definitions. Two nasty things can happen:
- You might overwrite manual changes you might have made to come of the code that was automatically generated.
- In the case of relational databases, Zendro code generators also create migrations (using Sequelize). As these are named using the current date, you might have several migrations to create the same tables. This will lead to errors. Make sure you delete the migrations folder content, if you want to run the code generators multiple times on the same model definitions:
rm ./graphql-server/migrations/*or run the codegen without the-mflag.
Upon starting the servers in any mode development or production any pending
database migrations and seeding is automatically triggered. See file
./graphql-server/migrateDb.js, ./graphql-server/startServer.js and the two docker-compose files
docker-compose-dev.yml (development) and docker-compose-prod.yml (production).
To configure the graphql-server create a .env inside the graphql-server folder (recommended) or set the variables via the docker-compose files.
It is mandatory to set the ALLOW_ORIGIN environment variable, which sets the Access-Control-Allow-Origin header, e.g.
ALLOW_ORIGIN="*"
For more details about the graphql-server environment variables, including the Keycloak/GraphiQL login setup below, see the graphql-server README
If you're using the zendro CLI's set-up/new commands (rather than this StarterPack's own yarn setup flow directly), a migration (2021-12-08T17_37_17.804Z#keycloak.js) runs automatically on first migrateDb.js up and sets up a default Keycloak realm, client, and admin user for you - see the console output it prints for the generated credentials. It writes the resulting OAUTH2_GRAPHIQL_CLIENT_ID/OAUTH2_GRAPHIQL_CLIENT_SECRET into graphql-server's own .env, and auto-generates a SESSION_SECRET there too if one isn't already set. graphiql-auth needs none of this itself - it holds no Keycloak credentials and only reverse-proxies to graphql-server, which is the sole service that talks to Keycloak. See graphql-server's own README, "GraphiQL & Authentication", for the full set of AUTH_*/OAUTH2_GRAPHIQL_*/SESSION_SECRET environment variables this depends on.
The single page application has to be aware of the URLs to use for login and to send GraphQL queries to. This is controlled by the following environment variables.
NEXT_PUBLIC_ZENDRO_GRAPHQL_URL='http://localhost:3000/graphql'NEXT_PUBLIC_ZENDRO_LOGIN_URL='http://localhost:3000/login'NEXT_PUBLIC_ZENDRO_EXPORT_URL='http://localhost:3000/export'NEXT_PUBLIC_ZENDRO_METAQUERY_URL='http://localhost:3000/meta_query'NEXT_PUBLIC_ZENDRO_MAX_UPLOAD_SIZE=500NEXT_PUBLIC_ZENDRO_MAX_RECORD_LIMIT=10000ZENDRO_DATA_MODELS='../data_model_definitions'
The recommended way is to create a .env.development and .env.production inside the single-page-app folder for your environment variables.`
Note that in case you are running the SPA via a docker container the ZENDRO_DATA_MODELS path has to reflect the location inside the container. If you are using the default dev|prod docker-compose files the folder will be mounted inside single-page-app.
For more details see our manual and the
single-page-applicationREADME.
Zendro can be used checking access rights for every single GraphQL query
received by the currently logged in user, identified by the Bearer token
(issued by Keycloak) found in the request header. The token is verified
against Keycloak's public key (OAUTH2_PUBLIC_KEY, set automatically by the
keycloak migration - see "GraphiQL login (Keycloak)" above) and the user's
roles are loaded from it to check access rights. This step is carried out by
the acl2 npm package. Respective
access rights can and must be declared in the file
./graphql-server/acl_rules.js
(generated from your data model definitions by the code generator).
You can run Zendro with or without this access control check. The default is to run it without checking access rights.
To switch access right checking on, pass the acl command line argument
when starting the server (node server.js acl), or uncomment the
//,'acl' line in
./graphql-server/startServer.js
if you're starting it via npm start/npm run dev.
Without access control, anyone able to reach /graphql can query and
mutate freely - GraphiQL itself doesn't need to be switched off separately
to restrict this (there's no such toggle in the current server.js); the
access control check above is what actually gates requests.
As long as you are developing your applications, you want the servers to react
to any changes you make to your code immediately. Hence, in the development
environment, the single-page-application is served through a dedicated server
and not compiled with webpack to be served statically.
yarn dev
Basically we now switch to production environment. The single-page-application
will be compiled with webpack and served statically.
yarn start
Have a look at the following examples, please.
If you want to generate a new Sequelize migration or seeder you need to do that
from within a Docker container created from the respective
zendro-graphql-server Docker image:
docker compose -f docker-compose-dev.yml run --rm zendro-graphql-server bash
./node_modules/.bin/sequelize seed:generate --name my_new_seeder
Note how we use docker-compose-dev.yml to have the local directory mounted
inside the Docker container, so that newly created files, like migrations or
seeder files, are actually persisted on the host file-system.
By default, application data is stored in a local SQLite file
(graphql-server/data.db) - see config/data_models_storage_config.json.
There's no separate application database container in the default
docker-compose-dev.yml/docker-compose-prod.yml setup to open a SQL
terminal to; the only Postgres container that ships by default
(zendro-keycloak-postgres) is Keycloak's own backing store, not your data.
To use a different storage backend (Postgres, MySQL, MongoDB, etc.) for your
own models, configure it in data_models_storage_config.json and add the
corresponding service to the docker-compose files yourself.
If you have started your docker-compose with -d or if you just want to delete the created containers, execute:
yarn stop dev|prod
The above [-dev] has to be used or not, depending on whether you ran the development or production environment.
To remove the docker images execute (see above):
docker images | grep zendrostarterpack | awk '{print "docker rmi " $1}' | sh
To delete the volumes permanently in which your data has been stored execute:
docker volume ls | grep zendrostarterpack | awk '{print "docker volume rm " $2}' | sh
Be warned: All your data will be lost!
If you also want to delete the Docker image holding the code generators execute:
docker rmi sciencedb-code-generators:latest
If you want to start from scratch, and generate the code for your model definitions again, we recommend to remove your local copies of graphql-server and single-page-application and check these sub-modules out again using git.
Zendro is the product of a joint effort between the Forschungszentrum Jülich, Germany and the Comisión Nacional para el Conocimiento y Uso de la Biodiversidad, México, to generate a tool that allows efficiently building data warehouses capable of dealing with diverse data generated by different research groups in the context of the FAIR principles and multidisciplinary projects. The name Zendro comes from the words Zenzontle and Drossel, which are Mexican and German words denoting a mockingbird, a bird capable of “talking” different languages, similar to how Zendro can connect your data warehouse from any programming language or data analysis pipeline.
Francisca Acevedo1, Vicente Arriaga1, Katja Dohm3, Constantin Eiteneuer2, Sven Fahrner2, Frank Fischer4, Asis Hallab2, Alicia Mastretta-Yanes1, Roland Pieruschka2, Alejandro Ponce1, Yaxal Ponce2, Francisco Ramírez1, Irene Ramos1, Bernardo Terroba1, Tim Rehberg3, Verónica Suaste1, Björn Usadel2, David Velasco2, Thomas Voecking3
- CONABIO - Comisión Nacional para el Conocimiento y Uso de la Biodiversidad, México
- Forschungszentrum Jülich - Germany
- auticon - www.auticon.com
- InterTech - www.intertech.de
Asis Hallab and Alicia Mastretta-Yanes coordinated the project. Asis Hallab designed the software. Programming of code generators, the browser based single page application interface, and the GraphQL application programming interface was done by Katja Dohm, Constantin Eiteneuer, Francisco Ramírez, Tim Rehberg, Veronica Suaste, David Velasco, Thomas Voecking, and Dan Wang. Counselling and use case definitions were contributed by Francisca Acevedo, Vicente Arriaga, Frank Fischer, Roland Pieruschka, Alejandro Ponce, Irene Ramos, and Björn Usadel. User experience and application of Zendro on data management projects was carried out by Asis Hallab, Alicia Mastretta-Yanes, Yaxal Ponce, Irene Ramos, Verónica Suaste, and David Velasco. Logo design was made by Bernardo Terroba.