15:02:51 <farheen_att> #startmeeting Acumos Architecture
15:02:51 <collabot`> Meeting started Thu Jan 10 15:02:51 2019 UTC.  The chair is farheen_att. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:51 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:02:51 <collabot`> The meeting name has been set to 'acumos_architecture'
15:03:58 <farheen_att> #info Manoop will host todays meeting.
15:07:29 <farheen_att> #topic Design Studio - ML Workbench Design
15:08:13 <farheen_att> #info Manoop: We discussed good points of why ML workbench is needed and now we want to see how ML Workbench will be integrated.
15:08:38 <farheen_att> #info Adi: I thought it would be a Q/A.  I shared this concept last year at a presentation.  I wasn't planning to use that.
15:08:57 <farheen_att> #info Kazi: I have the diagrams with me.  I will present and Adi can walk us through.
15:11:00 <farheen_att> #info Kazi: In November lockdown we have DS studio as one single tool.  We want to have multiple tools under DS umbrella.  To share data pipelines and other tools.
15:12:55 <farheen_att> #info Adi: Let me start by showing some vendors and how they are using.  It is critical for ML activities.  However when you are working on projects within the enterprise or general concepts of having assets linked together to have a construct of project.  Eclipse workspace where you have projects and assets.  CMLP and Acumos have linked these objects together and they don't have a organizing project.  There is no conce
15:14:43 <farheen_att> #info The ability to share things with other users by adding a project construct.  You can share within a team.  I am preaching to the choir.  These are pretty std. ml examples.  Let me show a construct of workbenches.  CMLP you can have multiple instances is a concept of a workbench.  An org that wants to manage all of there projects.  Sharing and collaboration.  Workbenches allow you the construct of organizing struct
15:16:57 <farheen_att> #info The users like to work within a project.  You have to be able to segregate them from the top.  That is the why of why this workbench was thought of.  The other aspect are notebooks that tend to be separate.  IBM workbench is analytical object.  Having analytical assets inside of workbench you have the key factors for sharing.  The difficult part is sharing data that is where workbench can control.  You can share t
15:18:21 <farheen_att> #info The notebooks need to be managed within a project.  project can contain more than one flow.  It's a complex mixture that sits inside of a project.  We had a nice discussion of composite model vs. composite solution.  That is where they are managed.  It is managed with a security model.
15:18:39 <farheen_att> #info Reuben: Are you versioning projects and how does that relate to the model?
15:19:07 <farheen_att> #info Adi: Kazi please show the project.
15:20:04 <farheen_att> #info Kazi: Where in ML workbench will it show up?  Design Studio will be replaced with ML Workbench.
15:21:50 <farheen_att> #info Adi: It turns UI into micro front end.  Because workbenches Omnicomponents as an iframe.  This ms can be embedded vs being a monolithic architecture.  pipelines and models are micro front ends.  You don't see it in this screen shots but there are omnicomponents embeded witihin the larger portals.
15:22:07 <farheen_att> #info Reuben: So we can use these panels in different places?
15:23:08 <farheen_att> #info Reuben are you saying what we're looking at is the UI for Acumos?  Adi: We are still working on it.  My point is that a project is a omnicomponent embedded in the portal.  You can embed them any where you like.
15:23:55 <farheen_att> #info Reuben: We already have a concept of models having versions.  Now were are going to be using the models with projects with versions.
15:25:52 <farheen_att> #info Adi: Let's take a use case.  A project is ongoing.  Let's say your working on churn effort.  You create a churn project.  In the first iteration you have 2 models then second iteration has 3 models.  Models are added and removed from that ecosystem.  Project is not temporal like PMT.  Project is an organizing construct that says we are managing.
15:27:09 <farheen_att> #info Reuben: Then there needs to be a formal release process.  So that means I need to create that version.  Like the difference between web design and in tspace where you have to hit a publish button where you can see a time stamp.  But there's a cost in terms of complexity.  Now instead of having an ongoing process you have a staged process.
15:28:38 <farheen_att> #info Kazi: A project is not a deployable entity.  Reuben: That's what makes it complicated.  Kazi: We have a solution within a project that gets deployed.  The project remains in my Acumos.  That tells me I had 2 models with one pipeline and in version 3 I had 3 models without data pipelines.
15:30:13 <farheen_att> #info Reuben: Yes, I agree but this is adding complexity to the user and I'm not sure they will adopt this idea.  Because this is not a corporate tool it is open.  Changes are coming through from all places and now you are aritfically creating a coherent project. vs. I just read about the classifier and I want to try it.  Now you have added overhead to see if a project version.
15:31:33 <farheen_att> #info Adi: We have AI ML catalog and you can have contributions to the catalog.  No problems.  If you want to modify it it allows you to collaborate as a team.  Now you have a portal and you can import those models.  It is a facade or an organizational draft.
15:32:40 <farheen_att> #info Reuben: I understand it better.  I do want to say that we are getting push back of why are you adding these features?  We don't really know what we need to build.   We are trying to create a platform where a lot of tools can be attached and show the value of it rather than just being cool feature.  What is the value?
15:33:31 <farheen_att> #info Bryan: In that vain.  There is a team construct that we don't have.  This is part of the bigger picture of training with a pipeline which we don't currently have.  If you're looking for rationals Adi and team can add it.
15:33:46 <farheen_att> #info Reuben: I'm not seeing why this is helping us do training.
15:36:02 <farheen_att> #info Bryan: This is the tool that can pull all the training together.  Manoop: but in this construct if a developer have to create a project? Sayee: You can share if you chose to.  Adi: Collaboration is not a simple concept it is very complex.  Let's say I want to use Sayee's notebook but he can't share it with me.  But if I want to use his code the platform has to allow me to use his code without his data.  One is imp
15:36:09 <farheen_att> #info construct of a project.
15:36:56 <farheen_att> #info Manoop: For a developer who is using a jupyter notebook and trained that model.  Can he use the pipeline integration? Adi: Pipelines are independent and so are notebooks.  Yes the can they are different.
15:37:24 <farheen_att> #info Bryan: We have not fully design how the jupyter notebooks are going to be integrated into the UI.
15:38:41 <farheen_att> #info Manoop: Concern: Team is confident to start development but not sure how to design.  Bryan: I am working on jupyter notebook.  I am analyzing that.
15:39:29 <farheen_att> #info Manoop:  We evaluated jupyter and how it is being done and we were told ML workbench will address.  We want to know what are those flows and how are they integrated into the pipeline?  Can these flows still support these scenarios?
15:41:22 <farheen_att> #info Kazi: With this ML workbench I am starting from ground zero.  1.  I want to create the python.  I create a project in my workbench using jupyter as my tool.  I will train that software with the data.  I will push the data from the data source to my pipeline using niffy.  I have created a private notebook attached a data source now I am training the model.  Once I have a model that is trained I will share it with m
15:44:23 <farheen_att> #info Kazi: I am doing everything under this project in a private space. Reuben: Yes, it explains it doesn't necessarily mean it's a good thing.  It makes it project centric and I don't believe that our users are that.  K: I can have collaborate.  Sayee: Projects are not restricting how projects are used.  Through a project you have a better chance of sharing.  There are advantages. Manoop: Major concern is that we are
15:45:53 <farheen_att> #info Adi: No project is optional.  Take any commercialized google amazon and you come to the catalog and use model you need an organizational construct.  If you are just sharing then yes you don't need it.  If you want a large group to use pipelines then how would you use them?
15:47:20 <farheen_att> #info: Reuben: Componentize pipelines like you do for the models.  Someone from a different place and background I have six data sources that can utilize the pipeline.  I want to plug the data pipeline (which I don't understand) I do my model training.  I pick those out like I do today.
15:50:00 <farheen_att> #info Adi: Pipeline by itself is not enough.  When you work in large groups there you have to manage through either projects or pipelines.  Sayee: when you have multiple models and then you have a versioning you need this.  Manoop: In this immediate release can we solve this pipeline for a single model instead of adding complexity of another layer.  Sayee: We are saying this is a bigger vision beyond Boreas.  We are goi
15:52:32 <farheen_att> #info od ultmate goal.  This is a good to bottom up approach.  Manoop: Do we agree this is a bottom up approach?  Sayee: Adi?  Adi: Componentizing this is a good delivery approach.  User comm. is made of university users who like to do models.  Then you have the enterprise users who want to understand what is that model going to be.  Enterprise have difficulty managing datasets outside of this platform which is a huge r
15:53:47 <farheen_att> #info Adi: this is hard for the users.  I've had experience.  This is not the same class of users as the phds creating models.  Yes, you can work independently project is organizational construct.  You don't have to make it required but the option should be there for the user.
15:55:09 <farheen_att> #info Adi: Overall model should be in place.  Nimish: Even if you large orgs. and their partners.  Modularity and clear separation of roles and responsibilities but I'm assuming a lot of this sits outside the project then we hae the project itself but it will not be tightly integrated into one environment.
15:56:13 <farheen_att> #info Reuben: We don't need to build the over arching design as a first step.  We can move towards it.  We have to make the features separable and integratable then we're moving in the right direction.  Nimish: Project is a temporal construct not permanent.
15:56:37 <farheen_att> #info Adi: Enterprise project is temporal.
15:57:45 <farheen_att> #info Sayee: Assets do persist that is used to create it.  Unless project.  Adi: Project can support a six month view.  Churn projects don't ever shut down.  Diff
16:01:19 <farheen_att> #info It's an open ended construct.  Nimish: You can say network planning is a project.  Adi: It's meta data that gets tagged to an asset.  It's literally a tag or labeling mechanism.  N: Does Acumos need to solve?  Adi: How would you organize it.  Marketplace will have the data as well.  What should marketplace be enabling?  Adi: We have a catalog of models is not enough that is why we have this discussion.
16:03:05 <farheen_att> #info Sayee: Data is not static data.  You have to have versioning.  How will you tie the versions together in a catalog. Kazi: today we have model catalog but tomorrow the catalog will include models notebooks data pipelines etc.
16:04:53 <farheen_att> #info Bryan: Maybe we should take one of these ideas and start building on the acumos platform.  You have to apply a new capability to the existing platform and how it should be done.  The notion of a team is fundamental to your vision?  Yes, B: the ability to configure that is another concept right? Yes B: Take some of these concepts and apply them to acumos. Manoop: We are past time folks.
16:05:23 <farheen_att> #action Manoop: Talk to PTLs regarding todays decision.
16:06:22 <farheen_att> #action Adi: Put together achievable goal for this release.  Which component?  getting one workbook integrated?  If you can put these and review we can start the development.  Put a minimum goal of what you target?
16:06:57 <farheen_att> #info Kazi: We are doing the behind the scenes data modeling.  Scoping has not been done yet.  We will attempt the scope question next week.
16:07:37 <farheen_att> #action Kazi Provide a minimum scope of development for this release.
16:08:00 <farheen_att> #endmeeting