15:04:47 <farheen_att> #startmeeting Architecture Committee
15:04:47 <collabot`> Meeting started Wed Dec  4 15:04:47 2019 UTC.  The chair is farheen_att. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:47 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:04:47 <collabot`> The meeting name has been set to 'architecture_committee'
15:05:53 <farheen_att> #topic Agenda
15:08:01 <farheen_att> #info Bryan - we are going to have to tweek the deployment design for Demeter.
15:09:35 <farheen_att> #info Bryan - We can't ship he logs around.  I created an epic under deployment project we talked about this morning.  We are going to have to deploy the solution the nginx and side beat side cars as a single pod.  The resource overhead in solutions are going to be higher.  we can optimize.  We either choose the expensive design approach or conform
15:09:35 <farheen_att> to the restrictions.
15:09:48 <farheen_att> #info Bryan the model, nginx, and filebeat.
15:10:12 <farheen_att> #topic Loose ends for the Clio maintenance release
15:10:33 <farheen_att> #info R models are not building ms correctly.  Phillippe, Guy, and team are looking into that.
15:12:31 <farheen_att> #topic Decide on Data description languages used inside acumos.  When we describe the services by a model we use protobuf.  It is efficient and puts a barrier to entry for people running their models.  Understand the difference of protobuf to serialize and protobuf to describe what the data types of the ms used are.
15:14:49 <farheen_att> #info ONAP integration revealed protobuf is not as proficient as others.  Nested lists are not supported in protobuf.  We added a raw mode.  Instead of understanding the datatypes we simply say it is a raw bytes.   It is up to the model for inputs and outputs.  That was sufficient for ONAP integration.
15:15:02 <farheen_att> #info Bryan - If you include a schema you can discover.
15:16:12 <farheen_att> #info Bryan -  the concept of a schema is a representation of how the attributes are going to be structured.  Protobuf is a medium.
15:17:32 <farheen_att> #info Reuben - we're using protobuf as a schema definition and also as a serialization tool.
15:18:15 <farheen_att> #info Guy - We use an alternative API or completely throw away schema.  I will have a more detailed discussion next week.
15:19:47 <farheen_att> #info If we want to connect two JSON schema you can not.  You can connect protobuf to protobuf.
15:20:00 <farheen_att> #info Reuben does that mean the whole library is not available to me?
15:20:43 <farheen_att> #info Guy yes, there could be either.  JSON and protobuf can't talk to each other.  The problem with ONAP is they use JSON.
15:21:27 <farheen_att> #info JSON schema and OpenAPI are widely accepted standards. We wanted efficiency with binary objects which we get from protobuf.
15:22:42 <farheen_att> #info Raw mode is better than etiher of the schemas we have now.  Having raw mode is useful even outside of the discussion of incorporating JSON schema.
15:23:46 <farheen_att> #info DCAE from ONAP has a philopsophy that all data types are named titles in a catalog.  In acumos data types are anonymous. 3 floats are 3 floats and can be connected with 3 floats.
15:34:15 <farheen_att> #info Reuben - We found a lot of different translators for JSON.
15:34:58 <farheen_att> #info Justin - There are hurdles with protobuf using acumos vs. server to server communication.
15:36:07 <farheen_att> #info Guy - We talked about building a language specific tool that would generate an interface for the protobuf file.  You can say here is protobuf.  Give me the tool I need to call the Acumos model like a simple function call.  If we could generate language by language.
15:36:40 <farheen_att> #info Guy for example build a protobuf and call an existing model that will generate the classes or the code for me then you would not need protobuf.
15:37:58 <farheen_att> #info if you had a tool that said download this interface to this class.  In python you would connect to the acumos instance with credentials and it would create a class which had methods that had simple method calls that handled the serialization of protobuf and deserialized and gave it back in native format.
15:38:14 <farheen_att> #info Bryan that is a fairly common problem.
15:38:41 <farheen_att> #info Sayee - You want a simple JSON file that you can train your model with.
15:39:09 <farheen_att> #info Guy - During the training the part you don't need protobuf for example, csv file.  No need to mention protobuf.
15:39:34 <farheen_att> #info The model updates daily.  Are we complicating it for the user?
15:39:42 <farheen_att> #info Why are we restricting users?
15:40:39 <farheen_att> #info Justin - CSV input is used in the running the model.  Conversion is required out of order, then you have to find the right library.  This is the burden to the app developer.
15:41:11 <farheen_att> #info it was intended to download the protobuf in the language.  Then the csv files are reasonably straight forward?
15:41:45 <farheen_att> #info Sayee - That is additional latency.  Why don't was support the format that developers want.
15:42:12 <farheen_att> #info we don't restrict everyone to protobuf.  you can send data in JSON as well.
15:43:00 <farheen_att> #info but if you want to isolate the data  transformation from the model part there are tools that will read from a file, table, and prepare them to feed into the acumos model.
15:43:09 <farheen_att> #info Why can't I use the data directly.
15:43:20 <farheen_att> #info What does latency mean in your context?
15:43:55 <farheen_att> #info For a real time onap network prediction in milliseconds and respond whether it's a fraud call or not are we going to take time.
15:44:30 <farheen_att> #info That needs to experimented.  Protobuf is very efficient.  Parsing the CSV file will take longer than Protobuf.
15:44:58 <farheen_att> #info For predicting the data marshalling.
15:45:28 <farheen_att> #info Sayee - Mojo and Pojo don't support protobuf.  Are we going to convert them from protobuf to CSV.
15:46:13 <farheen_att> #info Guy - I would not worry about the data marshaling time.
15:47:41 <farheen_att> #info Bryan - A model can take in any data type.  You hav to discover what that model is expecting or have a contract in advance such as meta data so the client knows.  You can send a request "what are you configured as?"  response is I am JSON or protobuf.  The model runner could support different types of input to the model.
15:48:10 <farheen_att> #info Do we want to implement as a separate piece or the modelrunner?
15:48:16 <farheen_att> #info modelrunner
15:48:24 <farheen_att> #info every modelrunner
15:49:03 <farheen_att> #info there are modeler runners for each language such as python model runner, r model runner, etc...
15:49:27 <farheen_att> #info That work has to be done for all of the model runner.
15:49:46 <farheen_att> #info Sayee - Maintenance of code becomes challenging.
15:50:18 <farheen_att> #info Yes, so we only have to write one model runner per language.
15:50:33 <farheen_att> #info It is possible we will defer this to E release.
15:51:56 <farheen_att> #info Justin - When I did the UI it was without he possibility of doing a JSON.  then the botteneck would be the conversion of the CSV file.
15:53:12 <farheen_att> #info Guy - Data frame type of work we do in protobuf is weird.  It's a transposed version of the way I like to think of the data.  You could conceive something that is malformed.  Needs work.
15:53:36 <farheen_att> #action Team post suggestions to the wiki.
15:53:59 <farheen_att> #info next week Bryan agreed to talk about namespace.
15:54:19 <farheen_att> #info Bryan - I did put my thoughts on the developer guide.
15:55:21 <farheen_att> #info Sayee next weeks topic I would like to add Glowroot to the agenda.
15:56:40 <farheen_att> #info Glowroot is a jar file that monitors your service account.  It gives memory dumps for data analysis.  It's a service.  It is for monitoring both platform.
15:57:13 <farheen_att> #info YOu can add it to a system as a common repo or to invidual component.
15:57:43 <farheen_att> #info we have Prometheus with our OS using ELK.  Bring the argument for that comparison.
15:58:47 <farheen_att> #info Sayee - How is training planned by Ericsson.  What is the model training approach taken.  Can we take into account Nifi.
15:59:23 <farheen_att> #info Bryan - I put an epic for deploying pipelines.  Training use cases and deploying pipelines needs to be discussed.
16:00:14 <farheen_att> #info Justin - For training we should be on the list.  We are not ready for next week.  We are not focused on Nifi at the moment.  We do have some ethics in plance under the training project.  Feedback will be helpful.
16:00:36 <farheen_att> #endmeeting