Skip to content

Add a script to upload reports to GitHub #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rhcu wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add a script to upload reports to GitHub #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4d84f34
WIP: Add a script to upload reports to GitHub
rhcu Aug 3, 2018
17c23e4
Add changes to github.py
rhcu Aug 3, 2018
fe82ec6
Created index.html that points to other reports
rhcu Aug 7, 2018
64fdddd
Merge branch 'master' into github_script
rhcu Aug 7, 2018
83fd144
Removed spaces
rhcu Aug 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion coverage_crawler/crawler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,4 +290,4 @@ def run(website):

driver.quit()

return os.path.abspath(os.path.join(os.getcwd(), '{}/report'.format(data_folder)))
return os.path.abspath(os.path.join(os.getcwd(), data_folder))
48 changes: 48 additions & 0 deletions coverage_crawler/github.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# -*- coding: utf-8 -*-

import os
import shutil
import subprocess


def upload_to_github(reports_list, git_user_name, git_password):
# Clone the repository if doesn't exist
if not os.path.isdir('coverage-crawler-reports'):
subprocess.run(['git', 'clone', 'https://github.com/rhcu/coverage-crawler-reports'])
os.chdir('coverage-crawler-reports')

# Remove the content of repository except of README
Copy link
Copy Markdown
Collaborator

@marco-c marco-c Aug 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the README too, the repository will only be accessed from https://rhcu.github.io/coverage-crawler-reports/, so the README is unneeded.

prev_files = os.listdir(os.getcwd())
for f in prev_files:
if os.path.isdir(f):
shutil.rmtree(f)
elif f.startswith('.') is False:
os.remove(f)
subprocess.run(['git', 'pull', 'https://github.com/rhcu/coverage-crawler-reports', 'master'])
Copy link
Copy Markdown
Collaborator

@marco-c marco-c Aug 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of cloning and pulling, we can do like we do here: https://github.com/mozilla/release-services/blob/master/src/shipit_code_coverage/shipit_code_coverage/github.py#L74 and just always overwrite the contents of the repository.

Copy link
Copy Markdown
Collaborator

@marco-c marco-c Aug 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So basically:

  1. create a coverage-crawler-reports directory;
  2. chdir to ^;
  3. run the crawler;
  4. create the index file;
  5. your current code from git init to the end.

You won't even need to move the reports this way, as they are created directly in the right directory.


with open('index.html', 'w') as f:
f.write("""
<html>
<head>
<title>Reports</title>
</head>
<body>
<h1>The list of available reports for websites:</h1>
<br>
""")
for website, report in reports_list.items():
# Push the new content
shutil.move(report, os.getcwd())
name_of_folder = report.rsplit('/', 1)[-1]
f.write('<a href="{}">{}</a>'.format(os.path.join(name_of_folder, 'report/index.html'), website))
f.write('<br>')
f.write("""
</body>
</html>
""")

# Commit the content
subprocess.run(['git', 'init'])
subprocess.run(['git', 'add', '*'])
subprocess.run(['git', 'commit', '-m', 'Coverage crawler reports upload'])
subprocess.run(['git', 'push', 'https://{}:{}@github.com/rhcu/coverage-crawler-reports'.format(git_user_name, git_password), 'master', '--force'])
6 changes: 6 additions & 0 deletions run_crawler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
# -*- coding: utf-8 -*-

from coverage_crawler import crawler
from coverage_crawler import github

reports = {}

with open('websites.txt') as f:
for website in f:
report = crawler.run(website)
reports[website] = report

github.upload_to_github(reports, 'GIT_ACCOUNT', 'GIT_PASSWORD')