Skip to content

Fix for misaligned data columns when exporting data from TXT to CSV using Python file #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hujambo-dunia wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix for misaligned data columns when exporting data from TXT to CSV using Python file #13

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions data/process_query_results.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
import json

LOCAL_DIRECTORY_READ = '../../psql-main-data/STEP_2_1_export_enumerate/' ## contains Txt files (created from SQL exports from AWS)
LOCAL_DIRECTORY_WRITE = '20221019/' ## contains Csv files (created from above Txt files)
# LOCAL_DIRECTORY_PREFIX = '202210181743/' ## unique date-time string value for multiple runs
LOCAL_DIRECTORY_WRITE = '202305/' ## contains Csv files (created from above Txt files)
# LOCAL_DIRECTORY_PREFIX = '202210181743/' ## unique date-time string value for multiple same-day runs

def get_tool_name(tool_id):
arr = tool_id.split('/')
Expand All @@ -28,7 +28,7 @@ def normalize_headers(columns, is_pivot):
if is_pivot:
columns.insert(1, "tool_id")
else:
columns.insert(0, "tool_id")
columns.insert(1, "tool_id")
return columns

## function that (1) exports the Txt file data into Csv files (2) removes null values (3) enumerates the rows in the Csv files (4) normalizes tool name/version
Expand Down