diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..afcc7ab4 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,51 @@ +# Architecture + +## High Level Overview + +The Security tracker consists of a server process that's responsible for +handling HTTP requests and a worker process that's doing all the background +processing. + +![Overall Architecture](./svg/architecture.svg) + +### External services + +The tracker needs to communicate with third party services, namely: + +1) The GitHub API for user authentication, team permissions, creating issues, etc +2) Two GitHub repos: + a) https://github.com/nixos/nixpkgs to pull the latest changes on nixpkgs + b) https://github.com/CVEProject/cvelistV5 to pull CVE data +3) https://prometheus.nixos.org/ to get information about the latest channels + + +# Database Schema + +![Database Schema](./svg/models.svg) + +# Nix Store, storage space and filesystem considerations + +The tracker requires significant storage space to run properly. In particular, +there are two different areas you'll need storage for, the database and the Nix +store. + +## Database + +In order to store three diffent Nixpkgs releases, you'll need around 80GB of +space for the Postgresql database. This is the case right now for storing +24.05, 24.11 and 25.05. + +As the number of CVEs and packages increases, the storage space requirement +will most likely increase as well. + +## Nix Store and filesystem + +Since the tracker builds derivations for all packages, it creates a lot of +small files in the filesystem. You need to make sure you have both enough space +for this and enough inodes on your filesystem. We suggest you [optimise the nix +store](https://wiki.nixos.org/wiki/Storage_optimization) and turn on automatic +optimisation. + +If you're using ext4, read this [Nix issue](https://github.com/NixOS/nix/issues/1522) +as you'll need to enable support for `large_dir`in your filesystem for +optimisation to work. diff --git a/docs/architecture.mermaid b/docs/architecture.mermaid new file mode 100644 index 00000000..31d30f75 --- /dev/null +++ b/docs/architecture.mermaid @@ -0,0 +1,67 @@ +graph TB + Users["**👥 Users**"] + + subgraph External["**External Services**"] + GitHub["**GitHub API**"] + GitHubNixos["**GitHub Repository**
*nixos/nixpkgs*"] + GitHubCVEs["**GitHub Repository**
*CVEProject/cvelistV5*"] + NixMonitoring["**NixOS Monitoring**
*Channel Status*"] + end + + subgraph SecurityTracker ["**Security Tracker Host**"] + subgraph Web["**Web**"] + Nginx["**Nginx HTTP**"] + WSGI["**WSGI Django**
*Django Views*"] + end + + subgraph ManageCommands["**Management Commands**"] + FetchAllChannels["**fetch_all_channels**"] + IngestCVEs["**ingest_delta_cve**"] + end + + subgraph Background["**Background Tasks**"] + SystemdTimerChannels["**Systemd Timer Fetch Channels**"] + SystemdTimerCVEs["**Systemd Timer Ingest CVEs**"] + NixEval["**Evaluate Nix**"] + DjangoWorker["**Django worker**"] + end + + subgraph Storage["**Storage**"] + PostgreSQL["**PostgreSQL**
*CVE Records
Channels
Users
Issues*"] + LocalGitCheckout["**Local Git Repo**
*nixpkgs clone /var/lib/web-security-tracker/nixpkgs-repo*"] + NixStore["**Nix store**"] + end + + end + + %% User interactions + Users -->|HTTP Request| Nginx -->|Forward| WSGI -->|Queries| PostgreSQL + WSGI --> GitHub + + %% Timers + SystemdTimerChannels -.->|**Triggers Daily**| FetchAllChannels + FetchAllChannels -->|1 Fetch Channels| NixMonitoring + FetchAllChannels -->|2 Git pull| GitHubNixos + FetchAllChannels -->|3 Update Repo| LocalGitCheckout + FetchAllChannels -->|4 Evaluate Nix| NixEval --> NixStore + + SystemdTimerCVEs -.->|**Triggers Daily**| IngestCVEs + IngestCVEs -->|1 Fetch CVEs| GitHubCVEs + IngestCVEs -->|2 Update Database| PostgreSQL + IngestCVEs -->|3 PgTrigger Suggestions| DjangoWorker + + classDef userClass fill:#e1f5fe,stroke:#01579b,stroke-width:3px,color:#000 + classDef externalClass fill:#f3e5f5,stroke:#4a148c,stroke-width:3px,color:#000 + classDef webClass fill:#e8f5e8,stroke:#2e7d32,stroke-width:3px,color:#000 + classDef commandClass fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#000 + classDef backgroundClass fill:#fce4ec,stroke:#c2185b,stroke-width:3px,color:#000 + classDef storageClass fill:#f1f8e9,stroke:#33691e,stroke-width:3px,color:#000 + classDef subgraphClass fill:#fafafa,stroke:#424242,stroke-width:3px + + class Users userClass + class GitHub,GitHubNixos,GitHubCVEs,NixMonitoring externalClass + class Nginx,WSGI webClass + class FetchAllChannels,IngestCVEs commandClass + class SystemdTimerChannels,SystemdTimerCVEs,NixEval,DjangoWorker backgroundClass + class PostgreSQL,LocalGitCheckout,NixStore storageClass + class Storage,Background,ManageCommands,Web subgraphClass diff --git a/docs/svg/architecture.svg b/docs/svg/architecture.svg new file mode 100644 index 00000000..96583402 --- /dev/null +++ b/docs/svg/architecture.svg @@ -0,0 +1 @@ +

Security Tracker Host

External Services

Storage

Background Tasks

Management Commands

Web

HTTP Request

Forward

Queries

Triggers Daily

1 Fetch Channels

2 Git pull

3 Update Repo

4 Evaluate Nix

Triggers Daily

1 Fetch CVEs

2 Update Database

3 PgTrigger Suggestions

👥 Users

GitHub API

GitHub Repository
nixos/nixpkgs

GitHub Repository
CVEProject/cvelistV5

NixOS Monitoring
Channel Status

Nginx HTTP

WSGI Django
Django Views

fetch_all_channels

ingest_delta_cve

Systemd Timer Fetch Channels

Systemd Timer Ingest CVEs

Evaluate Nix

Django worker

PostgreSQL
CVE Records
Channels
Users
Issues

Local Git Repo
nixpkgs clone /var/lib/web-security-tracker/nixpkgs-repo

Nix store

 \ No newline at end of file diff --git a/docs/svg/models.svg b/docs/svg/models.svg new file mode 100644 index 00000000..d1c1fde2 --- /dev/null +++ b/docs/svg/models.svg @@ -0,0 +1,3008 @@ + + + + + + +model_graph + + + +django_contrib_admin_models_LogEntry + + +     +    LogEntry     +     +id +     +     +AutoField +     +     +content_type +     +     +ForeignKey (id) +     +     +user +     +     +ForeignKey (id) +     +     +action_flag +     +     +PositiveSmallIntegerField +     +     +action_time +     +     +DateTimeField +     +     +change_message +     +     +TextField +     +     +object_id +     +     +TextField +     +     +object_repr +     +     +CharField +     + + + + +django_contrib_auth_models_User + + +     +    User +< +AbstractUser +>     +     +id +     +     +AutoField +     +     +date_joined +     +     +DateTimeField +     +     +email +     +     +EmailField +     +     +first_name +     +     +CharField +     +     +is_active +     +     +BooleanField +     +     +is_staff +     +     +BooleanField +     +     +is_superuser +     +     +BooleanField +     +     +last_login +     +     +DateTimeField +     +     +last_name +     +     +CharField +     +     +password +     +     +CharField +     +     +username +     +     +CharField +     + + + + +django_contrib_admin_models_LogEntry->django_contrib_auth_models_User + + + user (logentry) + + + +django_contrib_contenttypes_models_ContentType + + +     +    ContentType     +     +id +     +     +AutoField +     +     +app_label +     +     +CharField +     +     +model +     +     +CharField +     + + + + +django_contrib_admin_models_LogEntry->django_contrib_contenttypes_models_ContentType + + + content_type (logentry) + + + +django_contrib_auth_models_AbstractUser + + +     +    AbstractUser +< +AbstractBaseUser,PermissionsMixin +>     +     +date_joined +     +     +DateTimeField +     +     +email +     +     +EmailField +     +     +first_name +     +     +CharField +     +     +is_active +     +     +BooleanField +     +     +is_staff +     +     +BooleanField +     +     +is_superuser +     +     +BooleanField +     +     +last_login +     +     +DateTimeField +     +     +last_name +     +     +CharField +     +     +password +     +     +CharField +     +     +username +     +     +CharField +     + + + + +django_contrib_auth_base_user_AbstractBaseUser + + +   +AbstractBaseUser +   + + + +django_contrib_auth_models_AbstractUser->django_contrib_auth_base_user_AbstractBaseUser + + + abstract +inheritance + + + +django_contrib_auth_models_PermissionsMixin + + +   +PermissionsMixin +   + + + +django_contrib_auth_models_AbstractUser->django_contrib_auth_models_PermissionsMixin + + + abstract +inheritance + + + +django_contrib_auth_models_Permission + + +     +    Permission     +     +id +     +     +AutoField +     +     +content_type +     +     +ForeignKey (id) +     +     +codename +     +     +CharField +     +     +name +     +     +CharField +     + + + + +django_contrib_auth_models_Permission->django_contrib_contenttypes_models_ContentType + + + content_type (permission) + + + +django_contrib_auth_models_Group + + +     +    Group     +     +id +     +     +AutoField +     +     +name +     +     +CharField +     + + + + +django_contrib_auth_models_Group->django_contrib_auth_models_Permission + + + + permissions (group) + + + +django_contrib_auth_models_User->django_contrib_auth_models_AbstractUser + + + abstract +inheritance + + + +django_contrib_auth_models_User->django_contrib_auth_models_Permission + + + + user_permissions (user) + + + +django_contrib_auth_models_User->django_contrib_auth_models_Group + + + + groups (user) + + + +django_contrib_sessions_base_session_AbstractBaseSession + + +     +    AbstractBaseSession     +     +expire_date +     +     +DateTimeField +     +     +session_data +     +     +TextField +     + + + + +django_contrib_sessions_models_Session + + +     +    Session +< +AbstractBaseSession +>     +     +session_key +     +     +CharField +     +     +expire_date +     +     +DateTimeField +     +     +session_data +     +     +TextField +     + + + + +django_contrib_sessions_models_Session->django_contrib_sessions_base_session_AbstractBaseSession + + + abstract +inheritance + + + +allauth_account_models_EmailAddress + + +     +    EmailAddress     +     +id +     +     +AutoField +     +     +user +     +     +ForeignKey (id) +     +     +email +     +     +EmailField +     +     +primary +     +     +BooleanField +     +     +verified +     +     +BooleanField +     + + + + +allauth_account_models_EmailAddress->django_contrib_auth_models_User + + + user (emailaddress) + + + +allauth_account_models_EmailConfirmation + + +     +    EmailConfirmation     +     +id +     +     +AutoField +     +     +email_address +     +     +ForeignKey (id) +     +     +created +     +     +DateTimeField +     +     +key +     +     +CharField +     +     +sent +     +     +DateTimeField +     + + + + +allauth_account_models_EmailConfirmation->allauth_account_models_EmailAddress + + + email_address (emailconfirmation) + + + +allauth_socialaccount_models_SocialApp + + +     +    SocialApp     +     +id +     +     +AutoField +     +     +client_id +     +     +CharField +     +     +key +     +     +CharField +     +     +name +     +     +CharField +     +     +provider +     +     +CharField +     +     +provider_id +     +     +CharField +     +     +secret +     +     +CharField +     +     +settings +     +     +JSONField +     + + + + +allauth_socialaccount_models_SocialAccount + + +     +    SocialAccount     +     +id +     +     +AutoField +     +     +user +     +     +ForeignKey (id) +     +     +date_joined +     +     +DateTimeField +     +     +extra_data +     +     +JSONField +     +     +last_login +     +     +DateTimeField +     +     +provider +     +     +CharField +     +     +uid +     +     +CharField +     + + + + +allauth_socialaccount_models_SocialAccount->django_contrib_auth_models_User + + + user (socialaccount) + + + +allauth_socialaccount_models_SocialToken + + +     +    SocialToken     +     +id +     +     +AutoField +     +     +account +     +     +ForeignKey (id) +     +     +app +     +     +ForeignKey (id) +     +     +expires_at +     +     +DateTimeField +     +     +token +     +     +TextField +     +     +token_secret +     +     +TextField +     + + + + +allauth_socialaccount_models_SocialToken->allauth_socialaccount_models_SocialApp + + + app (socialtoken) + + + +allauth_socialaccount_models_SocialToken->allauth_socialaccount_models_SocialAccount + + + account (socialtoken) + + + +pgpubsub_models_Notification + + +     +    Notification     +     +id +     +     +AutoField +     +     +channel +     +     +CharField +     +     +created_at +     +     +DateTimeField +     +     +db_version +     +     +IntegerField +     +     +payload +     +     +JSONField +     + + + + +pghistory_models_Context + + +     +    Context     +     +id +     +     +UUIDField +     +     +created_at +     +     +DateTimeField +     +     +metadata +     +     +JSONField +     +     +updated_at +     +     +DateTimeField +     + + + + +pghistory_models_Events + + +     +    Events     +     +pgh_slug +     +     +TextField +     +     +pgh_context +     +     +JSONField +     +     +pgh_context_id +     +     +UUIDField +     +     +pgh_created_at +     +     +DateTimeField +     +     +pgh_data +     +     +JSONField +     +     +pgh_diff +     +     +JSONField +     +     +pgh_id +     +     +BigIntegerField +     +     +pgh_label +     +     +TextField +     +     +pgh_model +     +     +CharField +     +     +pgh_obj_id +     +     +TextField +     +     +pgh_obj_model +     +     +CharField +     + + + + +pghistory_models_MiddlewareEvents + + +     +    MiddlewareEvents     +     +user +     +     +ForeignKey (id) +     +     +url +     +     +TextField +     + + + + +pghistory_models_MiddlewareEvents->django_contrib_auth_models_User + + + user (middlewareevents) + + + +pghistory_models_MiddlewareEvents->pghistory_models_Events + + + proxy +inheritance + + + +shared_models_nix_evaluation_TimeStampMixin + + +     +    TimeStampMixin     +     +created_at +     +     +DateTimeField +     +     +updated_at +     +     +DateTimeField +     + + + + +pghistory_models_Event + + +     +    Event     +     +pgh_created_at +     +     +DateTimeField +     +     +pgh_label +     +     +TextField +     + + + + +shared_models_nix_evaluation_NixMaintainer + + +     +    NixMaintainer     +     +github_id +     +     +IntegerField +     +     +email +     +     +CharField +     +     +github +     +     +CharField +     +     +matrix +     +     +CharField +     +     +name +     +     +CharField +     + + + + +shared_models_nix_evaluation_NixLicense + + +     +    NixLicense     +     +id +     +     +BigAutoField +     +     +deprecated +     +     +BooleanField +     +     +free +     +     +BooleanField +     +     +full_name +     +     +CharField +     +     +redistributable +     +     +BooleanField +     +     +short_name +     +     +CharField +     +     +spdx_id +     +     +CharField +     +     +url +     +     +URLField +     + + + + +shared_models_nix_evaluation_NixSourceProvenance + + +     +    NixSourceProvenance     +     +id +     +     +BigAutoField +     +     +qualifier +     +     +CharField +     +     +source +     +     +BooleanField +     + + + + +shared_models_nix_evaluation_NixPlatform + + +     +    NixPlatform     +     +id +     +     +BigAutoField +     +     +system_double +     +     +CharField +     + + + + +shared_models_nix_evaluation_NixDerivationMeta + + +     +    NixDerivationMeta     +     +id +     +     +BigAutoField +     +     +available +     +     +BooleanField +     +     +broken +     +     +BooleanField +     +     +description +     +     +TextField +     +     +homepage +     +     +URLField +     +     +insecure +     +     +BooleanField +     +     +known_vulnerabilities +     +     +ArrayField +     +     +main_program +     +     +CharField +     +     +name +     +     +CharField +     +     +position +     +     +URLField +     +     +search_vector +     +     +SearchVectorField +     +     +unfree +     +     +BooleanField +     +     +unsupported +     +     +BooleanField +     + + + + +shared_models_nix_evaluation_NixDerivationMeta->shared_models_nix_evaluation_NixMaintainer + + + + maintainers (nixderivationmeta) + + + +shared_models_nix_evaluation_NixDerivationMeta->shared_models_nix_evaluation_NixLicense + + + + licenses (nixderivationmeta) + + + +shared_models_nix_evaluation_NixDerivationMeta->shared_models_nix_evaluation_NixSourceProvenance + + + + source_provenances (nixderivationmeta) + + + +shared_models_nix_evaluation_NixOutput + + +     +    NixOutput     +     +id +     +     +BigAutoField +     +     +output_name +     +     +CharField +     + + + + +shared_models_nix_evaluation_NixStorePathOutput + + +     +    NixStorePathOutput     +     +id +     +     +BigAutoField +     +     +store_path +     +     +CharField +     + + + + +shared_models_nix_evaluation_NixDerivationOutput + + +     +    NixDerivationOutput     +     +id +     +     +BigAutoField +     +     +derivation_path +     +     +CharField +     + + + + +shared_models_nix_evaluation_NixDerivationOutput->shared_models_nix_evaluation_NixOutput + + + + outputs (nixderivationoutput) + + + +shared_models_nix_evaluation_NixChannel + + +     +    NixChannel +< +TimeStampMixin +>     +     +channel_branch +     +     +CharField +     +     +created_at +     +     +DateTimeField +     +     +head_sha1_commit +     +     +CharField +     +     +release_version +     +     +CharField +     +     +repository +     +     +CharField +     +     +staging_branch +     +     +CharField +     +     +state +     +     +CharField +     +     +updated_at +     +     +DateTimeField +     + + + + +shared_models_nix_evaluation_NixChannel->shared_models_nix_evaluation_TimeStampMixin + + + abstract +inheritance + + + +shared_models_nix_evaluation_NixEvaluation + + +     +    NixEvaluation +< +TimeStampMixin +>     +     +id +     +     +BigAutoField +     +     +channel +     +     +ForeignKey (channel_branch) +     +     +attempt +     +     +IntegerField +     +     +commit_sha1 +     +     +CharField +     +     +created_at +     +     +DateTimeField +     +     +elapsed +     +     +FloatField +     +     +failure_reason +     +     +TextField +     +     +state +     +     +CharField +     +     +updated_at +     +     +DateTimeField +     + + + + +shared_models_nix_evaluation_NixEvaluation->shared_models_nix_evaluation_TimeStampMixin + + + abstract +inheritance + + + +shared_models_nix_evaluation_NixEvaluation->shared_models_nix_evaluation_NixChannel + + + channel (evaluations) + + + +shared_models_nix_evaluation_NixDerivation + + +     +    NixDerivation     +     +id +     +     +BigAutoField +     +     +metadata +     +     +OneToOneField (id) +     +     +parent_evaluation +     +     +ForeignKey (id) +     +     +attribute +     +     +CharField +     +     +derivation_path +     +     +CharField +     +     +name +     +     +CharField +     +     +search_vector +     +     +SearchVectorField +     +     +system +     +     +CharField +     + + + + +shared_models_nix_evaluation_NixDerivation->shared_models_nix_evaluation_NixDerivationMeta + + metadata (derivation) + + + +shared_models_nix_evaluation_NixDerivation->shared_models_nix_evaluation_NixStorePathOutput + + + + outputs (nixderivation) + + + +shared_models_nix_evaluation_NixDerivation->shared_models_nix_evaluation_NixDerivationOutput + + + + dependencies (nixderivation) + + + +shared_models_nix_evaluation_NixDerivation->shared_models_nix_evaluation_NixEvaluation + + + parent_evaluation (derivations) + + + +shared_models_cve_Organization + + +     +    Organization     +     +uuid +     +     +UUIDField +     +     +short_name +     +     +CharField +     + + + + +shared_models_cve_CveRecord + + +     +    CveRecord     +     +id +     +     +BigAutoField +     +     +assigner +     +     +ForeignKey (uuid) +     +     +requester +     +     +ForeignKey (uuid) +     +     +cve_id +     +     +CharField +     +     +date_published +     +     +DateTimeField +     +     +date_reserved +     +     +DateTimeField +     +     +date_updated +     +     +DateTimeField +     +     +local_timestamp +     +     +DateTimeField +     +     +serial +     +     +PositiveIntegerField +     +     +state +     +     +CharField +     +     +triaged +     +     +BooleanField +     + + + + +shared_models_cve_CveRecord->shared_models_cve_Organization + + + assigner (assigned) + + + +shared_models_cve_CveRecord->shared_models_cve_Organization + + + requester (requested) + + + +shared_models_cve_Product + + +     +    Product     +     +id +     +     +BigAutoField +     +     +vendor +     +     +CharField +     + + + + +shared_models_cve_SupportingMedia + + +     +    SupportingMedia     +     +id +     +     +BigAutoField +     +     +_type +     +     +CharField +     +     +base64 +     +     +BooleanField +     +     +value +     +     +TextField +     + + + + +shared_models_cve_Description + + +     +    Description     +     +id +     +     +BigAutoField +     +     +lang +     +     +CharField +     +     +search_vector +     +     +SearchVectorField +     +     +value +     +     +TextField +     + + + + +shared_models_cve_Description->shared_models_cve_SupportingMedia + + + + media (description) + + + +shared_models_cve_Tag + + +     +    Tag     +     +id +     +     +BigAutoField +     +     +value +     +     +CharField +     + + + + +shared_models_cve_Reference + + +     +    Reference     +     +id +     +     +BigAutoField +     +     +name +     +     +CharField +     +     +url +     +     +CharField +     + + + + +shared_models_cve_Reference->shared_models_cve_Tag + + + + tags (reference) + + + +shared_models_cve_ProblemType + + +     +    ProblemType     +     +id +     +     +BigAutoField +     +     +_type +     +     +CharField +     +     +cwe_id +     +     +CharField +     + + + + +shared_models_cve_ProblemType->shared_models_cve_Description + + + + description (problemtype) + + + +shared_models_cve_ProblemType->shared_models_cve_Reference + + + + references (problemtype) + + + +shared_models_cve_Impact + + +     +    Impact     +     +id +     +     +BigAutoField +     +     +capec_id +     +     +CharField +     + + + + +shared_models_cve_Impact->shared_models_cve_Description + + + + description (impact) + + + +shared_models_cve_Metric + + +     +    Metric     +     +id +     +     +BigAutoField +     +     +attack_complexity +     +     +CharField +     +     +attack_vector +     +     +CharField +     +     +availability_impact +     +     +CharField +     +     +base_score +     +     +FloatField +     +     +base_severity +     +     +CharField +     +     +confidentiality_impact +     +     +CharField +     +     +format +     +     +CharField +     +     +integrity_impact +     +     +CharField +     +     +privileges_required +     +     +CharField +     +     +raw_cvss_json +     +     +JSONField +     +     +scope +     +     +CharField +     +     +user_interaction +     +     +CharField +     +     +vector_string +     +     +CharField +     + + + + +shared_models_cve_Metric->shared_models_cve_Description + + + + scenarios (metric) + + + +shared_models_cve_Event + + +     +    Event     +     +id +     +     +BigAutoField +     +     +description +     +     +ForeignKey (id) +     +     +time +     +     +DateTimeField +     + + + + +shared_models_cve_Event->shared_models_cve_Description + + + description (event) + + + +shared_models_cve_Credit + + +     +    Credit     +     +id +     +     +BigAutoField +     +     +description +     +     +ForeignKey (id) +     +     +user +     +     +ForeignKey (uuid) +     +     +_type +     +     +CharField +     + + + + +shared_models_cve_Credit->shared_models_cve_Organization + + + user (credit) + + + +shared_models_cve_Credit->shared_models_cve_Description + + + description (credit) + + + +shared_models_cve_Platform + + +     +    Platform     +     +id +     +     +BigAutoField +     +     +name +     +     +CharField +     + + + + +shared_models_cve_Version + + +     +    Version     +     +id +     +     +BigAutoField +     +     +less_equal +     +     +CharField +     +     +less_than +     +     +CharField +     +     +status +     +     +CharField +     +     +version +     +     +CharField +     +     +version_type +     +     +CharField +     + + + + +shared_models_cve_Cpe + + +     +    Cpe     +     +id +     +     +BigAutoField +     +     +name +     +     +CharField +     +     +search_vector +     +     +SearchVectorField +     + + + + +shared_models_cve_Module + + +     +    Module     +     +id +     +     +BigAutoField +     +     +name +     +     +CharField +     + + + + +shared_models_cve_ProgramFile + + +     +    ProgramFile     +     +id +     +     +BigAutoField +     +     +name +     +     +CharField +     + + + + +shared_models_cve_ProgramRoutine + + +     +    ProgramRoutine     +     +id +     +     +BigAutoField +     +     +name +     +     +CharField +     + + + + +shared_models_cve_AffectedProduct + + +     +    AffectedProduct     +     +id +     +     +BigAutoField +     +     +collection_url +     +     +CharField +     +     +default_status +     +     +CharField +     +     +package_name +     +     +CharField +     +     +product +     +     +CharField +     +     +repo +     +     +CharField +     +     +search_vector +     +     +SearchVectorField +     +     +vendor +     +     +CharField +     + + + + +shared_models_cve_AffectedProduct->shared_models_cve_Platform + + + + platforms (affectedproduct) + + + +shared_models_cve_AffectedProduct->shared_models_cve_Version + + + + versions (affectedproduct) + + + +shared_models_cve_AffectedProduct->shared_models_cve_Cpe + + + + cpes (affectedproduct) + + + +shared_models_cve_AffectedProduct->shared_models_cve_Module + + + + modules (affectedproduct) + + + +shared_models_cve_AffectedProduct->shared_models_cve_ProgramFile + + + + program_files (affectedproduct) + + + +shared_models_cve_AffectedProduct->shared_models_cve_ProgramRoutine + + + + program_routines (affectedproduct) + + + +shared_models_cve_Container + + +     +    Container     +     +id +     +     +BigAutoField +     +     +cve +     +     +ForeignKey (id) +     +     +provider +     +     +ForeignKey (uuid) +     +     +_type +     +     +CharField +     +     +date_assigned +     +     +DateTimeField +     +     +date_public +     +     +DateTimeField +     +     +search_vector +     +     +SearchVectorField +     +     +source +     +     +JSONField +     +     +title +     +     +CharField +     + + + + +shared_models_cve_Container->shared_models_cve_Organization + + + provider (container) + + + +shared_models_cve_Container->shared_models_cve_CveRecord + + + cve (container) + + + +shared_models_cve_Container->shared_models_cve_Description + + + + descriptions (container) + + + +shared_models_cve_Container->shared_models_cve_Description + + + + configurations (container_configurations) + + + +shared_models_cve_Container->shared_models_cve_Description + + + + workarounds (container_workarounds) + + + +shared_models_cve_Container->shared_models_cve_Description + + + + solutions (container_solutions) + + + +shared_models_cve_Container->shared_models_cve_Description + + + + exploits (container_exploits) + + + +shared_models_cve_Container->shared_models_cve_Tag + + + + tags (container) + + + +shared_models_cve_Container->shared_models_cve_Reference + + + + references (container) + + + +shared_models_cve_Container->shared_models_cve_ProblemType + + + + problem_types (container) + + + +shared_models_cve_Container->shared_models_cve_Metric + + + + metrics (container) + + + +shared_models_cve_Container->shared_models_cve_Event + + + + timeline (container) + + + +shared_models_cve_Container->shared_models_cve_Credit + + + + credits (container) + + + +shared_models_cve_Container->shared_models_cve_AffectedProduct + + + + affected (container) + + + +shared_models_cve_CveIngestion + + +     +    CveIngestion     +     +id +     +     +BigAutoField +     +     +delta +     +     +BooleanField +     +     +timestamp +     +     +DateTimeField +     +     +valid_to +     +     +DateField +     + + + + +shared_models_cve_NixpkgsIssue + + +     +    NixpkgsIssue     +     +id +     +     +BigAutoField +     +     +description +     +     +ForeignKey (id) +     +     +code +     +     +CharField +     +     +created +     +     +DateField +     +     +status +     +     +CharField +     + + + + +shared_models_cve_NixpkgsIssue->shared_models_nix_evaluation_NixDerivation + + + + derivations (nixpkgsissue) + + + +shared_models_cve_NixpkgsIssue->shared_models_cve_CveRecord + + + + cve (nixpkgsissue) + + + +shared_models_cve_NixpkgsIssue->shared_models_cve_Description + + + description (nixpkgsissue) + + + +shared_models_cve_NixpkgsEvent + + +     +    NixpkgsEvent     +     +id +     +     +BigAutoField +     +     +issue +     +     +ForeignKey (id) +     +     +reference +     +     +TextField +     + + + + +shared_models_cve_NixpkgsEvent->shared_models_cve_NixpkgsIssue + + + issue (nixpkgsevent) + + + +shared_models_cve_NixpkgsAdvisory + + +     +    NixpkgsAdvisory     +     +id +     +     +BigAutoField +     + + + + +shared_models_cve_NixpkgsAdvisory->shared_models_cve_NixpkgsIssue + + + + issues (nixpkgsadvisory) + + + +shared_models_linkage_CVEDerivationClusterProposal + + +     +    CVEDerivationClusterProposal +< +TimeStampMixin +>     +     +id +     +     +BigAutoField +     +     +cve +     +     +ForeignKey (id) +     +     +created_at +     +     +DateTimeField +     +     +status +     +     +CharField +     +     +updated_at +     +     +DateTimeField +     + + + + +shared_models_linkage_CVEDerivationClusterProposal->shared_models_nix_evaluation_TimeStampMixin + + + abstract +inheritance + + + +shared_models_linkage_CVEDerivationClusterProposal->shared_models_cve_CveRecord + + + cve (derivation_links_proposals) + + + +shared_models_CVEDerivationClusterProposalStatusEvent + + +     +    CVEDerivationClusterProposalStatusEvent +< +Event +>     +     +pgh_id +     +     +AutoField +     +     +pgh_context +     +     +ForeignKey (id) +     +     +pgh_obj +     +     +ForeignKey (id) +     +     +pgh_created_at +     +     +DateTimeField +     +     +pgh_label +     +     +TextField +     +     +status +     +     +CharField +     + + + + +shared_models_CVEDerivationClusterProposalStatusEvent->pghistory_models_Context + + + pgh_context (+) + + + +shared_models_CVEDerivationClusterProposalStatusEvent->pghistory_models_Event + + + abstract +inheritance + + + +shared_models_CVEDerivationClusterProposalStatusEvent->shared_models_linkage_CVEDerivationClusterProposal + + + pgh_obj (status_events) + + + +shared_models_linkage_MaintainersEdit + + +     +    MaintainersEdit     +     +id +     +     +BigAutoField +     +     +maintainer +     +     +ForeignKey (github_id) +     +     +suggestion +     +     +ForeignKey (id) +     +     +edit_type +     +     +CharField +     + + + + +shared_models_linkage_MaintainersEdit->shared_models_nix_evaluation_NixMaintainer + + + maintainer (maintainersedit) + + + +shared_models_linkage_MaintainersEdit->shared_models_linkage_CVEDerivationClusterProposal + + + suggestion (maintainers_edits) + + + +shared_models_linkage_DerivationClusterProposalLink + + +     +    DerivationClusterProposalLink     +     +id +     +     +BigAutoField +     +     +derivation +     +     +ForeignKey (id) +     +     +proposal +     +     +ForeignKey (id) +     +     +provenance_flags +     +     +IntegerField +     + + + + +shared_models_linkage_DerivationClusterProposalLink->shared_models_nix_evaluation_NixDerivation + + + derivation (derivationclusterproposallink) + + + +shared_models_linkage_DerivationClusterProposalLink->shared_models_linkage_CVEDerivationClusterProposal + + + proposal (derivationclusterproposallink) + + + +shared_models_DerivationClusterProposalLinkEvent + + +     +    DerivationClusterProposalLinkEvent +< +Event +>     +     +pgh_id +     +     +AutoField +     +     +derivation +     +     +ForeignKey (id) +     +     +pgh_context +     +     +ForeignKey (id) +     +     +pgh_obj +     +     +ForeignKey (id) +     +     +proposal +     +     +ForeignKey (id) +     +     +id +     +     +BigIntegerField +     +     +pgh_created_at +     +     +DateTimeField +     +     +pgh_label +     +     +TextField +     +     +provenance_flags +     +     +IntegerField +     + + + + +shared_models_DerivationClusterProposalLinkEvent->pghistory_models_Context + + + pgh_context (+) + + + +shared_models_DerivationClusterProposalLinkEvent->pghistory_models_Event + + + abstract +inheritance + + + +shared_models_DerivationClusterProposalLinkEvent->shared_models_nix_evaluation_NixDerivation + + + derivation (+) + + + +shared_models_DerivationClusterProposalLinkEvent->shared_models_linkage_CVEDerivationClusterProposal + + + proposal (+) + + + +shared_models_DerivationClusterProposalLinkEvent->shared_models_linkage_DerivationClusterProposalLink + + + pgh_obj (events) + + + +shared_models_cached_CachedSuggestions + + +     +    CachedSuggestions +< +TimeStampMixin +>     +     +proposal +     +     +OneToOneField (id) +     +     +created_at +     +     +DateTimeField +     +     +payload +     +     +JSONField +     +     +updated_at +     +     +DateTimeField +     + + + + +shared_models_cached_CachedSuggestions->shared_models_nix_evaluation_TimeStampMixin + + + abstract +inheritance + + + +shared_models_cached_CachedSuggestions->shared_models_linkage_CVEDerivationClusterProposal + + proposal (cached) + + + +webview_models_Profile + + +     +    Profile     +     +id +     +     +BigAutoField +     +     +user +     +     +OneToOneField (id) +     + + + + +webview_models_Profile->django_contrib_auth_models_User + + user (profile) + + + +webview_models_Profile->shared_models_cve_NixpkgsIssue + + + + subscriptions (subscribers) + + + diff --git a/infra/README.md b/infra/README.md index afc04f7c..576be616 100644 --- a/infra/README.md +++ b/infra/README.md @@ -37,6 +37,6 @@ Then commit `secrets/name_of_secret.age` as usual. ## Metrics -A [Prometheus Node Exporter](https://github.com/prometheus/node_exporter) is running exposing host specfic metrics. These are scraped by the nixos.org [Prometheus](https://prometheus.nixos.org/graph) and are also available under [Grafana](https://grafana.nixos.org/). +A [Prometheus Node Exporter](https://github.com/prometheus/node_exporter) is running exposing host specfic metrics. These are scraped by the nixos.org [Prometheus](https://prometheus.nixos.org/graph) and are also available under [Grafana](https://grafana.nixos.org/d/rYdddlPWk/node-exporter-full?orgId=1&from=now-24h&to=now&timezone=browser&var-datasource=default&var-job=node&var-node=tracker.security.nixos.org:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B&refresh=1m). -Moreover, a [Postgres Exporter](https://github.com/prometheus-community/postgres_exporter) and a [SQL exporter](https://github.com/justwatchcom/sql_exporter) run in the host exposing Postgres and application specific metrics. +Moreover, a [Postgres Exporter](https://github.com/prometheus-community/postgres_exporter) and a [SQL exporter](https://github.com/justwatchcom/sql_exporter) run in the host exposing Postgres and application specific metrics. A dashboard for these metrics is [here](https://grafana.nixos.org/d/beo2uotj65lvkb/nix-security-tracker?orgId=1&from=now-6h&to=now&timezone=browser&var-Instance=tracker.security.nixos.org:9237).