Support query protocol (SNS, SQS, EC2 and more)

[dstendardi]

Currently, the aws-codegen supports JSON and JSON-rest based services. In order to bring supports for older services, the query protocol has to be implemented.

Query protocol will bring support for services such as * s3, ec2, SNS, SQS.

Moving parts

A quick looks at the other languages SDK shows that there is two mains parts involved in this development :

• query serializer : responsible for transforming parameters objects into a query strings according to shape definition.
  As an example, we can refers to the javascript's query_param_serializer

• response deserializer : takes an xml response and transforms it into the matching erlang/elixir structure.
  As @kingoftheknoll noticed in this issue, xml responses does not contains attributes, which makes easy to convert the response to an elixir map.

Erlang and Elixir support

This repository seems to supports both erlang and elixir code generation. As @jkakar says in this issue it would be nice to have the serde code written in erlang, so it can be used in both aws-elixir and aws-erlang

Todo

• query serializer
• response deserializer
• query (elixir)
• query (erlang)

[dstendardi]

Finally managed to get some time to progress 😄. I did a first draft on the query serializer.
Basically it's a port of the nodeJS SDK. It still misses specific conditionals regarding the EC2 protocol. I chose to write it in elixir first because I feel confortable with it. If everything works as expected, i will consider writing it in erlang so we can share code between erlang and elixir generated libs.
Any feedback is warmly welcomed !

[dstendardi]

@jkakar what do you think about using https://github.com/homanchou/elixir-xml-to-map.
Seems very thin wrapper around erlsom doing exactly what we want (convert back xml response to elixir map).

@dstendardi looks like elixir-xml-to-map doesn't do any type casting and keeps everything as strings. Could be misreading it though.

One thing that's nice about sweet-xml is that it can process as lazy seqs and in some cases, take snapshots for example, the response can be huge. Some of our enterprise clients have 100k+ snapshots. Public images is the same way.

[dstendardi]

hello @kingoftheknoll (tried to ping you before but was not able until you join the issue 😋).
That's right it's really a dumb ass conversion. I was planning to do static typing in both request and response in another issue, but it might also be considered as a part of this one. If i had to do static typing, I would go for generating structures for requests and response from api specs.
I would also need to generate the sweet xml definition from the api spec.
Can you explain a bit about snapshots ?

Sorry for not being available. Yeah I would go that route.

I only mentioned lazy evaluation with sweet xml https://github.com/kbrw/sweet_xml#sweetxml-element-streaming because for certain resources types, depending on the account size, can be quite large. My company harvests AWS api endpoints and for large enterprise customers the number of resources in a response especially snapshots can be a huge number. So being able to parse the xml doc lazily seems like a big win in using sweet-xml. Just something to think about.

[jkakar]

@dstendardi This looks like a great start, thank you! It'd be lovely to see an Erlang version we can use to add support for both Erlang and Elixir, but either way, whatever you choose will be a nice step forward from where we are now.

[dstendardi]

hey @jkakar just tried adding erlang in the project and it works like a charm ! So I'am very looking forward learning the lower layer 😉 and will focus on porting the previous code directly to erlang !

[dstendardi]

Ported the query serializer to erlang. Since it's my first step in Erlang, I'am open to any early feedback :-).

For the query deserialization i plan to go for a first draft with a naive map deserialization which means no streaming (yet) but type handling. I think I will use the following nodejs parser tests as an example

Once I finish with the serde, I may try to introduce static typing with case class in elixir along with case formatting following the language conventions.

IE in elixir :

%ListTopicsResponse{next_token: "whatever"} =  sns.list_topics(%ListTopicsRequest{next_token: "whatever"})

@jkakar do you prefer sticking with AWS case formatting (maybe easier for new user to learn) or to apply case formatting (more idomatic) ?

[dstendardi]

hey ! Did some progress on the deserialization of xml response ! You can check the work here. @kingoftheknoll it's a very naive parser without streaming, but it relays on xmerl, so no extra dependency, and it's a first step. I really think adding streaming would bring some extra complexity I can't handle right now.

[jkakar]

@dstendardi I have a few things going on right now, so I won't get a chance to look at what you've done for at least a couple of days, but thank you for pushing this forward, I appreciate it. Anyway, it's on my todo list, I'll get here as quickly as I can.

[dstendardi]

don't worry, @jkakar, I have enough work right now with the interoperability (ie : Erlang support for binaries, and date conversions). I just write things here to keep track of the progress.

I also cleaned up my todo list, in order to make the PR smaller :-)

[dstendardi]

I am now working on the generated service layer in elixir and I have a few questions 😀

• looks like error shapes are not properly defined in the specification files (eg: all definitions contains a "message" property which should be named "Message". So I decided to mimic the go sdk which seems to do some naive deserialization ignoring the metadata. So I am planning to do the same and make the deserializer return a tuple of code, message, and requestId. Does it seems ok to you ?

• Since I have external files in erlang, I am also thinking about copying this files to the target repository. Regarding this constraint, maybe it would be nice to extract the utility functions (which are generic) from the template, and cover it with some unit testing ?

• I also converted all the charlist types to binary types inside the xml deserializer in order to have good interop between erlang and elixir. Is it fine ?

[jkakar]

That all sounds great! I vaguely remember some issues with error messages that led to the current design, but maybe they're resolved now (doing whatever the SDK does is right, IMO).