#acumos-meeting: Architecture Committtee

Meeting started by farheen_att at 14:02:59 UTC (full logs).

Meeting summary

  1. Licensing (farheen_att, 14:04:45)
    1. Working on things that are difficult to demo because we still need the front end pieces. We created the api to interact with the portal. We have handed over what portal can deliver in their parts. LUM is what we demonstrated on the Community Call yesterday. It isn't a great user demo but we will review the enhancements that have been (farheen_att, 14:07:10)
    2. cut short demo for on-boarding team offshore who have to leave early. (farheen_att, 14:07:55)

  2. On-boarding Priya, Guy Spark Java Spark on-boarding (farheen_att, 14:08:37)
    1. https://wiki.acumos.org/display/MOB/Acumos+spark (farheen_att, 14:11:38)
    2. Priya today we are assuming that you are using the Spark ML library. (farheen_att, 14:13:01)
    3. with respect to spark lib. Does the lib have to be packaged as a k8 container? (farheen_att, 14:13:29)
    4. yes (farheen_att, 14:13:32)
    5. current ms gen method we put everything inside the container. (farheen_att, 14:14:11)
    6. so every container will contain the spark lib? (farheen_att, 14:14:22)
    7. it will be packaged as part of the jar (farheen_att, 14:14:50)
    8. the model docker image will be independent of the docker image. (farheen_att, 14:15:03)
    9. so I can have a spark cluster outside for my model to run? (farheen_att, 14:15:16)
    10. correct (farheen_att, 14:15:20)
    11. regarding mode of deployment there are two ways client spark and cluster mode. We are considering the standalone spark for this version of implementation. The apis used are different based on which manager is being used as part of the deployment. (farheen_att, 14:16:57)
    12. we don't really know which one should be used and we wanted to verfiy based on configuration. YML or K8. Today we are focusing on stand alone deployment of spark. (farheen_att, 14:18:01)
    13. YARN not YML* (farheen_att, 14:19:02)
    14. being able to onboard spark models. But for Clio we are limiting it to java spark based on boarding. (farheen_att, 14:19:35)
    15. Java spark based model will remain as a docker image where a modeler will deploy? (farheen_att, 14:20:06)
    16. it is the same process and ms generation is the same as any other type of model. (farheen_att, 14:20:25)
    17. how does it differ? (farheen_att, 14:20:52)
    18. the apis used are different. Whatever apis used to invoke the instances are different. (farheen_att, 14:21:11)
    19. there are two types of libraries available today. and Mlib and a spark MLIB. Available from spark 2.3 onwards. (farheen_att, 14:22:41)
    20. 2 ways to create docker image. 1. python based models today we are deploying all the keras dependencies contained. But in the case of spark it is going to be outside the docker image. The docker won't have the docker engine only the spark libs to talk to spark instance. (farheen_att, 14:24:13)
    21. important to have a clear input output. Spark ML models may be reading from a hadoop cluster or a different data source as of today. Spark ML model will define a clear input and output per protobuf. (farheen_att, 14:25:22)
    22. whatever required has to be specified. (farheen_att, 14:25:38)
    23. can we not have a single container for model is consuming resoures and performance will hit. (farheen_att, 14:26:16)
    24. agree the spark engine will be independent. (farheen_att, 14:26:46)
    25. don't be tightly coupled with protobuf (farheen_att, 14:27:11)
    26. long term we should involve. JSON and Parkay file support should be there. If you look at customers they are using other file formats. My recommendation is to keep it generic. (farheen_att, 14:28:28)
    27. We want to support both protobuf and JSON. (farheen_att, 14:28:57)
    28. for now we will support only protobuf but eventually we will support other types. v.1 protobuf and v.2 JSON and then others. (farheen_att, 14:30:07)
    29. these are the apis that model runner will support. Invoking the model will have a sequence when you invoke the model. The spark config that you have can be different than what is in the deployed format. Accepting and submitting the config at run time. Other models are self contained but in term of spark since the engine is outside the (farheen_att, 14:32:16)
    30. Data source and output are both coming from the same response. It is independent of what fs to use. These are consideration to invoke the model. Docker image will be submitting the job but the input output will be outside of request/response. (farheen_att, 14:33:21)
    31. this is the view we thought that kafka and spark to go together. We will need a pipeline for kafka to support spark. (farheen_att, 14:34:20)
    32. java client is impacted by this change. (farheen_att, 14:35:01)

  3. Licensing resume. (farheen_att, 14:35:18)
    1. Michelle, Justin (farheen_att, 14:35:28)
    2. We've completed the apis for license profile. We can now review the apis for this sprint. (farheen_att, 14:36:15)
    3. we're starting with 3336 which is an api manager. In this demo we are using junit test. We are using different json files checked in as resources. A license validator and the warnings. Once you upload your license profile. You will be able to call this api and get feedback vs. down stream problems. This is the back end of reading the (farheen_att, 14:39:32)
    4. as far as different variables missing. This is the validation of the structure of the document and the unit test. When it does pass as the license json. When you call this from the api perspective we wrap it under the profile class. 3 inputs. Give it a string, input stream, or json node if you've done the parsing of the json for (farheen_att, 14:41:17)
    5. questions? (farheen_att, 14:41:24)
    6. Michelle we showed the schema validation api to make sure it is captured. (farheen_att, 14:41:47)
    7. Please open the response? So you will have multiple messages are there any status? We will check the field if it's a success we won't do anything. (farheen_att, 14:43:05)
    8. tausif - I thought we will be calling for onboarding but there is a different flow for just the license file so you want portal to call when there is a license. (farheen_att, 14:45:31)
    9. yes we want to catch issues up front. (farheen_att, 14:45:41)
    10. will there be an option to edit online or do they have to reload the file? (farheen_att, 14:45:55)
    11. yes, they have to upload againg. The editor uses the schema. Upload without using editor. The editor is baked into (farheen_att, 14:46:55)
    12. is the format json? (farheen_att, 14:47:02)
    13. yes (farheen_att, 14:47:07)
    14. stored as blob in? (farheen_att, 14:47:16)
    15. nexus (farheen_att, 14:47:20)
    16. can it be hacked? (farheen_att, 14:47:28)
    17. only the system component. User does not access nexus (farheen_att, 14:47:53)
    18. to create a custom license requirement you have to ready readthedocs. (farheen_att, 14:51:32)
    19. We made sure that all of our java apis are updated to java 11. We were able to update. (farheen_att, 14:52:12)
    20. you use the docker base images? (farheen_att, 14:52:20)
    21. we help the portal team update to open j9, It wasn't licensing specific. There were issues. There are two libs that licensing manager client library and we helped them with the LUM code. In order to interface with LUM there is a code generator that connects with LUM. LMCL is consuming that . I wanted to show you that. It wasn't a jira. (farheen_att, 14:58:17)
    22. demonstrated how to register the software with the LUM with the solution id. Reaches out to CDS and nexus to gather the information based on the solution id. (farheen_att, 14:59:07)
    23. We're trying to simplify how clients interface with LUM. Construct swid tag (revision id) so the mapping is consisitent in the LCML. (farheen_att, 15:00:00)
    24. demonstrated the library working as a bonus (farheen_att, 15:00:13)
    25. ACTION: Murali co-ordinate a full end to end demo with portal. In your scrum or their scrum have a joined session. (farheen_att, 15:05:47)
    26. ACTION: Manoop add license review to the next call. (farheen_att, 15:06:39)


Meeting ended at 15:06:45 UTC (full logs).

Action items

  1. Murali co-ordinate a full end to end demo with portal. In your scrum or their scrum have a joined session.
  2. Manoop add license review to the next call.


People present (lines said)

  1. farheen_att (69)
  2. collabot` (3)


Generated by MeetBot 0.1.4.