Geo id and the storing of trace data

[dylanjmcconnell]

For the isp-trace-parser we stored data as either demand, project or zone types.

The project vs zone essentially maps delineates between "existing" (e.g. Existing Committed Anticipated Additional , ECAA) projects, at a specific location and potential ("extendable") projects (i.e. projects that can be build in a capacity expansion model) - within a particular zone (area, rather than point source).

Within the ISPyPSA we have introduced the concept of a geo id that flattens the distinction between zones/areas and actually existing projects/ECAA point sources. On the one hand, geo id is not a term explicitly used AEMO, but on the other - they don't actually distinguish between project and zone like we do (and implicitly use a general geo id).

One potential side benefit is that it avoids problems identified in issue #36 - namely that some of the "REZ's" are now actually "ISP subregions" (or "Distrubtion REZ's").

So I propose flattening the underlying data structure to have (say) demand and resource (or similar).

*Could potentially flatten further, so simple trace flat files, but lose some access/query efficiency given the difference in columns between demand and resource traces data.

[dylanjmcconnell]

Some more concrete details:

Current data schema:

Essentially three tables, as follows:

Demand:

Column	Type
datetime	timestamp[us]
value	double
subregion	string
poe	string
demand_type	string
scenario	string (partition)
reference_year	int32 (partition)

Where:

• demand_type= OPSO_MODELLING', OPSO_MODELLING_PVLITE , or PV_TOT
• poe=POE10, POE50
• scenario = Green Energy Exports, Progressive Change, Step Change

Project:

Column	Type
datetime	timestamp[us]
value	double
project	string
resource_type	string
reference_year	int32 (partition)

Where:

• resource_type = FFP, SAT, WIND

Zone:

Column	Type
datetime	timestamp[us]
value	double
zone	string
resource_type	string
reference_year	int32 (partition)

Where:

• resource_type = CST, SAT, WFL, WFX, WH, WM

Potential new schema:

Essentially merge the zone / project tables - and reference only a geoid:

Column	Type
datetime	timestamp[us]
value	double
geo_id	string
resource_type	string
reference_year	int32 (partition)

Where:

• resource_type = FFP, SAT, WIND , CST, WFL, WFX, WH, WM

(nb. resource_type here could now be a bit more confusing (e.g. that SAT is in both existing project and zone data, where as wind exists as WIND for existing, and WH/HW for prospective sites).

A third meta option

Merge everying thing, one flat schema:

Column	Type
datetime	timestamp[us]
value	double
geo_id	string
trace_type	string
reference_year	int32 (partition)

Where:

• resource_trace = everything (and likely merge names categories where necessary - e.g. "Step Change POE 10 OPSO Modelling`

[dylanjmcconnell]

One thing would note about the meta schema - (probably not the right word) - flagged above, is that it is (I think?) pretty robust to expansion overtime. As / if new trace types emerge (or new scenarios, or other categories) - can easily append to the table.

(On the flip side, could potentially hide all manner of sins in there as well. Not advocating for this per se).

[dylanjmcconnell]

See below for some examples of the proposed schema for the resource mapping (which is keyed by the IASR ID from the IASR workbook)

 # Existing named generator
  "Bango 973 Wind Farm":
    resource_type: wind
    geo_id: BANGOWF1
    rez:  Null #Some will be associated with REZ, some not
    subregion: SNSW
    region: NSW
    trace_names: [BANGOWF1]

# REZ traces
  "Q1_SAT_Far North QLD":
    resource_type: SAT
    geo_id: Q1
    rez: Q1  
    subregion: NQ
    region: QLD
    trace_names: [Q1_SAT_Far North QLD]

# Distributed REZ trace
  "NNSW SAT - Distributed Resources":
    resource_type: SAT
    geo_id: NNSW
    rez: Null #These are not associated with a REZ / only subregion
    subregion: NNSW
    region: NSW
    trace_names: [Distributed_Resources_NNSW]

Some interrelated questions:

• Is the geo_id strictly necessary in this mapping? The more generic geo_id is basically either the name of the project, the rez id for rez's and the subregion for distributed rez traces - all of which is also stored in the same schema. If this is genuinely something that might be uses as filter / selector (i.e. we want to select by these generic geo_id's then I think fine to keep) else can drop? (which doesn't mean needs to be dropped in the the ISPyPSA)

• Is this metadata necessary / good idea? (or should there be more?). For example, regions derive from subregion, rez derive from subregion. I think being explicit good - (particularly with Nulls for some / incomplete mapping) - but does introduce a little redundancy.

• Alternatively could add more (e.g. geo_type or similar, - "rez" , "project" (for ECAA), "region" (for Distributed Resources ). Maybe could drop geo_id and have this instead? (also not sure if this is useful filter or not)

[nick-gorman]

I think my main concern is the mixed ways of using geo_id. In ISPyPSA the geo_id of plant is the rez/subregion/region it is located in. However, here geo_id of a plant is its own name. Three ways I can see of clarifying this:

• Think of all plants in a rez/subregion/region as being located at just at that rez/subregion/region, not at distinct location with in the region or zone. Therefore there geo_id should be the region or zone, in both ispypsa and the trace parser.
• Think of plants as having there own distinct locations with in a region or zone. In which case what you propose for the trace parser makes sense, but ispypsa would need to change to use parent_geo_id (but see previous convos on parent terminology).
• Drop the use of geo_id in the trace parser, so we don't need to unify the concept across packages.

[dylanjmcconnell]

Thanks Nick - I hadn't really realised the difference with how geo_id is used in ISPyPSA - (.. and afaik we had proposed / discussed the geo_id as flattening the across zones, areas and existing projects/ECAA point sources? - see earlier in discussion. Maybe I have got that wrong).

A clarifying question I would have is how this works for the case of multiple plants having the same geo_id? For example, there would be for example (say) a dozen solar projects in a single REZ - which would all have the same geo_id, but would actually be different / point geographical locations ... perhaps if just flattening back to REZ, the geo_id concept makes less sense?

I think really the key think to understand / get a handle of is common / useful access patterns - ie. how the data is most likely / commonly going to be filtered and selected. When constructing the full ispypsa model, you likely want a full reference year or data for all resources (?) - but perhaps you want to select by technology, region, subregion, rez, individual trace, or all of the above.

[nick-gorman]

Your right, your previous comments in the discussion do clearly propose using geo_id to flatten across s zones, areas and existing projects/ECAA point sources, I just hadn't clocked that this was conflicting with how ISPyPSA was proposing to use geo_id.

Your right, if geo_id is the same for multiple plants you can't flatten on it. Could you instead flatten on id (or name / the key in your proposed resource mapping schema)?? Are the keys the resouce mapping schema the IASR names?

I think the access pattern from the ISPyPSA perspective is just to use the IASR names and handle the mapping to rez, subregion, region internally.

[dylanjmcconnell]

Yeah cool .. and yeah, the IASR ID is unique and the proposed key of the yaml mapping (though includes non-geo info - e.g. Q1_SAT_Far North QLD)

I think the current geo_id might kind of do what you propose? (unless I have misunderstood - as in, it flattens on IASR name, but just the geo component of the meta data - specific locations, or rez, or subregion)

Sounds like the key access pattern might be the IASR name / id - so maybe best to stick to that - unless some version of geo-id is helpful (.. keen to hear Ellie's thoughts) . Just a clarification, on using IASR names - would you handle the resource mapping internally as well (ie to solar , wind etc)?

[EllieKallmier]

ok have had a proper look through the traces and the IASR IDs and think I understand this discussion above so far - here's my thoughts out loud:

1. My understanding of the geo_id term is kind of rooted in the ISPyPSA use, which is that it's used there to simplify how locational information for assets is stored/defined. In ISPyPSA the geo_id for a generator for example is just either the REZ or subregion in which the generator is defined. Then we can use another table to filter/map between different locational granularity but it keeps the big summary tables cleaner.
2. The IASR ID is very handy and simplifies a bunch of stuff because it's unique for every generator, and for new entrants it contains both geographical (rez/subregion id) and resource type info. In terms of mapping between ISPyPSA generators and traces, it makes sense that this is the value to use, and yeah makes the geo_id thing redundant in the trace parser.
3. Re: your last clarification there Dylan, yep that's what I'd assume. We currently use the fuel_type (which will be "Wind" or "Solar") value in ISPyPSA to map at that high level, and now the IASR ID contains resource type info already for new entrants.
4. I just have one uncertainty/question - it doesn't look like there's a unique trace for each unique generator in the workbook (makes sense). Then my question is do the mappings for those generators without traces get set up here in the trace parser? (Or do we need to think about the metadata needed in the ISPyPSA generator info needed to map/filter to the right trace?)

I think my summary thought is basically in the trace parser I don't think the geo_id term is useful/necessary (from a mapping with ISPyPSA perspective). And yeah maybe just causes more confusion or weirdness to try and force it here...

[dylanjmcconnell]

Thanks Ellie! Very helpful and makes sense to me..

Re: point 4 - if I understand correctly, this (real) example below is an example of this? Two different IASR ID's (WLWLSF1 and WLWLSF2) map to the same trace name (Walla_Walla)

# Many-to-one: two IASR generator entries share a single trace file
"WLWLSF1":
    geo_resolution: generator
    resource_type: SAT
    subregion: SNSW
    region: NSW
    rez: Null
    trace_names: [Walla_Walla]

"WLWLSF2":                                                                                                                                                                                                                          
    geo_resolution: generator
    resource_type: SAT                                                                                                                                                                                                          
    subregion: SNSW
    region: NSW
    rez: Null
    trace_names: [Walla_Walla]

Let me know if that makes sense / answers your question?

In anycase, it sounds like the the metadata stored here is largely useful for third party users (i.e. users that want to use the trace parser library / data, but not with ISPyPSA - as most of that mapping will be handled internally to ISPyPSA anyway. (This basically relates back to #41 - where I think we decided it makes sense to including the mapping in both locations etc). But practically, the only mapping that would be directly consumed by ISPyPSA is the IASR-name to trace-name mapping - please let me know if have got that wrong!

[EllieKallmier]

Yes that does answer my question! That's how I assumed it would probably work but yeah wanted to be sure.

Also yeah that's my understanding re: handling mapping internally - now we can use the IASR ID for everything (previously we had to construct a kind of equivalent ID from the resource type/rez/project name etc) it's quite straightforward. So yes just the IASR-name to trace-name mapping is all the ISPyPSA will need!

**The only thing that I'm not 100% sure about is with the regional granularity stuff and whether that would mean anything different needed in the future - but again, the IASR ID for each gen will still be attached to equivalent 'metadata' re: location/resource type in the ISPyPSA tables so I don't think so!