The person in charge of the IT department requires a team of Data Engineers to carry out the design, development and presentation of software that allows working with a set of data. In the same way, these can be presented on a board that facilitates tactical decision-making in the medium and long term.
The objectives to be achieved in this project are the following:
- Develop a python application that performs an ETL starting from a plain text file (.csv) to a relational database (PostgreSQL)
- Develop an API with a graphical interface for the use of the python application that performs ETL
- Develop functionalities that allow transformations, ABM and queries on a relational database
- Develop tests on the developed python application
- Design and develop a dashboard in MicroStrategy that consumes data from a relational database
Our initial dataset corresponds to 6 csv format files:
- cities.csv: Information of the cities
- customer.csv: Customer information
- prod_cat_info.csv: Product information by category
- store_types.csv: Information about the types of stores
- stores.csv: Store information
- transactions.csv: Transaction information
- Analisis:
- EDA
- Dataset
- Insight
- Buster-Block:
- api
- logs
- data_client
- database
- docs
- source
- utils
- config
- db
- logs
- etl
- logs
- logs
- test
- webapp
- .streamlit
- pages
- uils_pages
- api
- Python > 3.7.0 Descargar Python
- PostgreSQL > 10.0.0 Descargar Docker
Important!
.env -> Edit the database credentials
Go to the working directory
Open command console
docker-compose up -d
python -m venv env
env/scripts/activate
pip install -r requirements.txt
python Buster-Block/api/api.py
Open new terminal
env/scripts/activate
cd Buster-Block/webapp
streamlit run home.py
Go to the working directory
Open command console
docker-compose up -d
python3 -m venv env
suorce env/bin/activate
pip install -r requirements.txt
python3 Buster-Block/api/api.py
Open new terminal
suorce env/bin/activate
cd Buster-Block/webapp
streamlit run home.py