#acumos-meeting: Architecture Committee

Meeting summary

1. Open Design decisions on Open workbench and training (farheen_cefalu_a, 15:06:22)
   1. Kazi: I have presented the APIs from the ML workbench side. Chris: has annotated the jira issues with Mvp that will be delivered in boreas timeframe. I put them in gerrit but have not received any comments. CDS is ready based on user stories. These are listed in jira? The CDS files are in RST. It would take a lot of time to review on this call. (farheen_cefalu_a, 15:12:22)
   2. Let's start with Kazi's api. Kazi: We have 5 apis. project, pipeline, and notebook service. I will go through the project service apis in some details. for the rest of the apis i'll go through critical api definitions. (farheen_cefalu_a, 15:14:36)
   3. Bryan: Review the number of emails to the list that are impacting the architecture. Bring it to peoples attention to make sure people are commenting. (farheen_cefalu_a, 15:16:08)
   4. Kazi: this is an ML workbench UI. All services will be docker based ms. Project, Notebook, and pipeline services will be stored in the back end CDS, jupyterhub, notebook, and nifi servers. I will be reviewing the empty blue box apis. (farheen_cefalu_a, 15:18:29)
   5. There are 5 apis for each of the service like create,update, list, delete (archive) the apis. With all the apis i have defined different sections. What are the paths. What triggers the api. What does the gui do and also the backend business logic which I call behavior. REST API Path Post /mlWorkbench/. This API si called when the user request the creation of a new Project in his ML Workbench worspace UI. (farheen_cefalu_a, 15:21:03)
   6. When this API is successfully executed a named project must be created. The new project will appear in the new project catalog as well as the acumos bell icon. User can open a project. If the api fails the user will receive a notification. Output is json format. The json format has already been discussed with the development team and Vaibhav will share. The backend will make sure the json is valid otherw (farheen_cefalu_a, 15:23:07)
   7. the uid is there. Also, the project name is mandatory. Project name may only contain alpha-numeric characters. Project version is optional. Check if the project name and version already exists for that user. If not the user receives a message. We are creating a new workbench. We are creating an entry in the user table. We are going to populate the authenticated user id. It will generat a new uuid. I (farheen_cefalu_a, 15:25:16)
   8. to create the artifacts in the dbase. The CDS must return a boolean. Create a new entry in the project table. We're going to populate the appropriate fields, id, identifier, version, owner. Chris: Are there open jira issues? Kazi: I will check. Chris: Check if these apis already exist. Kazi: I will share this document with you in the Training project. (farheen_cefalu_a, 15:27:35)
   9. ACTION: Kazi put the Training API document to the wiki. (farheen_cefalu_a, 15:27:53)
   10. Manoop: If you get back to your diagram of list of apis (power point). Who will be consuming these apis? ML Workbench. Who is developing the apis? Vaibhav and team. UI layer is Sayee. Sayee: Is this following the existing standard of APIs. (farheen_cefalu_a, 15:29:59)
   11. Sayee: I'm asking is there a pattern that Acumos follows? We follow the REST conventions. We make remote procedure calls and the path indicates the functions. We follow the standard conventions. Other than ML Workbench UI will there be any other components consuming these apis? Kazi: Not in Boreas. (farheen_cefalu_a, 15:32:02)
   12. We don't need to go into every endpoint. Sayee: I want to bring up the Nifi and Jupyter apis. They may fail based on Bryan's feedback. Some of these apis may change as we evolve. Kazi: Let me clear I am discussing the north bound APIs. The two bold black lines may change. Bryan: We have to conform the apis to jupyter and nifi as a project and they may change over time. Sayee: Also authenticate ID is not (farheen_cefalu_a, 15:34:36)
   13. Kazi: The authenticated user id is the login id. Why is it not obvious to use what we already have. S: How are the cookies being passed? ARe we going to make a call to get the id. Mukesh: I think it is part of a jenkins job to create a nifi node. S: When I click on created project how do i get the id. JWtoken that is using. Passing this id to jupyter is not possible. Not to an external system. (farheen_cefalu_a, 15:36:39)
   14. Bryan: We also have to work out how we're going to manage the nifi containers. We need a nifi hub and there is no hub for jupyter. Sayee: With respect to cluster each node is technically not assigned to a user. B: A nifi cluster is about spreading a training process across nodes. S: There are two aspects design time nad content. How do we manage the containers? We need a hub. It's an open question. We (farheen_cefalu_a, 15:38:41)
   15. Bryan: I added that to the end of the architecture diagram. (farheen_cefalu_a, 15:39:14)
   16. K: These are the details that Sayee, Bryan, and I reviewed. M: no open questions. the UI developers who will be consuming the apis have to ask questions. (farheen_cefalu_a, 15:40:42)
   
   
2. Mukesh Training Project Structures (farheen_cefalu_a, 15:41:51)
   1. WE will use angular, js along with polymer. The first option the node modules are checked in at the project level. The approach that we will get out of this be able to check out the project. There are maven tools that we are going to be using. (farheen_cefalu_a, 15:43:17)
   2. M: ML workbench UI infrastructure for the ui. M: isn't this node module folder obvious? M: what is the advantage. This will be a pattern that we will export to the other components to start this. Manoop: I do not see the pro as an actual pro. You are saying no need to pre-install (farheen_cefalu_a, 15:45:22)
   3. Mukesh: You can include the mpm cammands from maven. The node package has to be installed on the local pc. Concern we need to have a node package in the individual pcs. It is for both development and user. Manoop: Yes, for developer but for production isn't a concern because your module will be packaged as a docker image. (farheen_cefalu_a, 15:47:32)
   4. Manoop: We have done that in many other open source projects. They provide us access to run the mpm runtime commands so when the jenkins job kicks in mpm install. Maven, Jenkins, and fully automated fully install is the std. way everyone is doing in OpenSource. Mukesh: I will recommend instead of running the mvn modules at shell command. (farheen_cefalu_a, 15:49:28)
   5. Mukesh: We want to maintain the dependencies. We should use package.json. Mukesh: Understood. The developer needs node package manager installed in maven to make a clean build. Mukesh: We will have and existing folder Workbench-fe like polymer component. We would like to move forward. Sayee: We want to be as pluggable as possible. (farheen_cefalu_a, 15:51:17)
   6. https://gerrit.acumos.org/r/#/admin/projects/workbench (farheen_cefalu_a, 15:52:15)
   7. Sayee: The project catalog contains projects. Mukesh: project catalog is a separate docker container. Sayee: the project container will have how to modify individual projects. So this serves a UI? Yes. It is not user specific? It can be user specific. Bryan: A project is a a separate container that runs a subset of ui. It is a project container per platform. (farheen_cefalu_a, 15:55:53)
   8. Mukesh: It is better to have one UI that has service. Bryan and Mukesh are proposing they should be together. Manoop: I agree with Mukesh and Bryan. I don't see an advantage in keeping them separate. We are better off keeping them together. Sayee: that 's the goal. As a developer it is easier to update small containers vs. larger containers. Bryan: This is important architecture decision. Chris: YOu t (farheen_cefalu_a, 15:58:35)
   9. Mukesh: We have to plug them into a logical unit. It's much easier for developer. Kazi: I want to make sure that a pipeline catalog and a notebook catalog can exist outside of a project. Sayee: Agrees. K: As long as we can create pipeline and notebook outside the catalogs I am fine. (farheen_cefalu_a, 16:00:28)
   
   
3. Bryan reviewing acumos-architecture-detail-boreas.png v.6. (farheen_cefalu_a, 16:01:22)
   1. Bryan: Flink is a new service. These are all for deployments. We are going to be using sidecar for licenses tracking. These are in flight and I'm capturing in diagram. ML workbench fe and be implement all the web components. (farheen_cefalu_a, 16:03:52)
   2. Reminder to folks to use the development discussions in the devdiscuss. Manoop: Have a close discussions with PTLs about their individual components and then bring it to the committees. B: Get the CMLP to give a demo to the acumos project. ML workbench is one example. That will be essential for ptls to consider. What is the best code base. (farheen_cefalu_a, 16:06:09)

Action items

1. Kazi put the Training API document to the wiki.

People present (lines said)

1. farheen_cefalu_a (32)
2. collabot` (3)