07:59:59 <morgan_orange> #startmeeting Functest weekly meeting October 4th
07:59:59 <collabot> Meeting started Tue Oct  4 07:59:59 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:59:59 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
07:59:59 <collabot> The meeting name has been set to 'functest_weekly_meeting_october_4th'
08:00:12 <morgan_orange> #topic call role
08:00:21 <Cristina__> #info Cristina Pauna (enea)
08:00:47 <jose_lausuch> #info Jose Lausuch
08:01:08 <morgan_orange> #info Morgan Richomme
08:01:50 <morgan_orange> #info agenda https://wiki.opnfv.org/display/functest/Functest+Meeting
08:01:56 <morgan_orange> any point you want to add?
08:02:03 <hideyasu_ool> #info Hideyasu ool
08:02:23 <morgan_orange> #topic action points follow-up
08:02:33 <morgan_orange> #info AP1: morgan_orange push presentation on dahsboard in the doc repo
08:02:38 <morgan_orange> #info done https://gerrit.opnfv.org/gerrit/#/c/22659/
08:02:44 <morgan_orange> #info AP2: morgan_orange formalize PTL election in wiki (vote planned on the 10th so procedure shall be available 1 week before)
08:02:51 <morgan_orange> #info done https://wiki.opnfv.org/display/functest/Functest+admin
08:03:04 <morgan_orange> feel free to comment, election will be organized next week
08:03:13 <morgan_orange> #info AP3: jose_lausuch viktor_t lhinds May-meimei ollivier SerenaFeng juhak Raghav complete etherpad postmortem
08:03:21 <morgan_orange> #link https://etherpad.opnfv.org/p/Colorado-Testing-postmortem
08:03:27 <morgan_orange> #info I just see some update from yardstick, we will discuss Colorado postmortem in a topic
08:03:35 <jose_lausuch> sorry, didnt put anything new..
08:03:38 <morgan_orange> #info AP4: hideyasu_ool jose_lausuch sync with SDNVPN
08:03:44 <morgan_orange> #info done clarification done by hideyasu (mail)
08:03:51 <jose_lausuch> done, not many similarities
08:03:52 <morgan_orange> #info AP5: morgan_orange contact hongbo to see articulation dovetail tests / functest tests (especially in the VIM section)
08:04:02 <morgan_orange> #info done hongbo confirmed they will not redevelop test cases but base their suite on existing suite (wiki was a bit misleading for me)
08:04:17 <morgan_orange> any comment on the actions points of the week?
08:05:06 <morgan_orange> #topic Colorado post mortem
08:05:38 <morgan_orange> maybe we can plan a slot during OpenStack summit to finalize and let some time to complete
08:05:45 <jose_lausuch> yes
08:05:50 <jose_lausuch> I;ll be attending
08:05:53 <morgan_orange> there are already several remarks
08:06:28 <morgan_orange> #action jose_lausuch morgan_orange plan finalization of Colorado postmortem during OpenStack Summit
08:06:35 <jose_lausuch> what shall we do with this?
08:06:38 <morgan_orange> #action everybody 2 more weeks to complete...
08:06:43 <jose_lausuch> go point by point and comment?
08:07:02 <morgan_orange> I would suggets to formalize a presentation that we could share with other testing projects/feature projects
08:07:06 <jose_lausuch> I mean during the review
08:07:07 <jose_lausuch> ok
08:07:33 <jose_lausuch> for the "-" we should have proposals for improvement
08:07:35 <morgan_orange> organize a little bit the points and suggest improvement for Danube for the - and the --
08:07:38 <morgan_orange> yes
08:07:52 <jose_lausuch> :)
08:08:02 <morgan_orange> #action morgan_orange jose_lausuch initiate presentation with proposals for the - and -- alreday referenced
08:08:37 <morgan_orange> but we encourage everybody, especially the peopel who contribute to Colorado to share their remark/criticism/pain points...
08:08:55 <morgan_orange> do not be shy....
08:09:01 <jose_lausuch> for sure
08:09:18 <morgan_orange> #topic Danube: Discussion on internal tescases
08:09:18 <jose_lausuch> for any topic (code, docs, apis,...)
08:09:47 <morgan_orange> yes maybe we should distinguish framework, tests (unit/internal, features), code, project management,..
08:10:07 <morgan_orange> as we have in the Danube page
08:10:15 <morgan_orange> #link https://wiki.opnfv.org/display/functest/Functest+Danube
08:10:42 <morgan_orange> we have several types of tests: unit, internal, feature, vnf
08:11:24 <jose_lausuch> I think we need to strengthen the unit testing in opnfv
08:11:41 <jose_lausuch> for our own functions
08:11:46 <jose_lausuch> own tooling
08:11:48 <morgan_orange> at least the commodity functions
08:11:52 <jose_lausuch> yes
08:11:58 <morgan_orange> yes agree, there is already a JIRA for that
08:12:07 <jose_lausuch> and they could be used for verify jobs for ex.
08:12:12 <jose_lausuch> ah ok
08:12:18 <jose_lausuch> didnt see it yet, sorry
08:12:52 <morgan_orange> #info unit tests should be integrated especially for our own functions
08:12:54 <morgan_orange> #link https://jira.opnfv.org/browse/FUNCTEST-336
08:13:04 <morgan_orange> we also suggest an intership
08:13:19 <morgan_orange> first step will consist in unit testing the functions in util directory
08:13:19 <jose_lausuch> for that task?
08:13:49 <morgan_orange> yes but it is not the season for interns at the moment
08:14:12 <jose_lausuch> ya
08:14:14 <jose_lausuch> ok
08:14:15 <morgan_orange> #info intership proposed on Functest unit tests
08:14:18 <morgan_orange> #link https://wiki.opnfv.org/display/DEV/Intern-projects-page
08:14:28 <morgan_orange> #link https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests
08:14:35 <morgan_orange> maybe we shall start..
08:14:55 <jose_lausuch> we can clarify in the summit
08:15:02 <jose_lausuch> what to do exactly
08:15:08 <jose_lausuch> for wich exact tools
08:15:15 <morgan_orange> #info new internal tests have also been mentioned: evolution of vPing, vPing + real userdata (not using local drive), security groups
08:15:44 <jose_lausuch> sec groups would be a new test case
08:15:52 <jose_lausuch> already proposed for colorado
08:16:08 <jose_lausuch> do you know if david would have time to look into it?
08:16:23 <jose_lausuch> because I think its a very good idea
08:16:29 <morgan_orange> #info Dovetail defined a vRouter test case that shoudl also be not too difficult to implement
08:16:52 <morgan_orange> for the moment we collect the idea, but we will need to allocate our resources for Danube...
08:17:08 <morgan_orange> note this vRouter has nothing to deal with vRouter vnf from ool
08:18:04 <morgan_orange> do you see other interesting internal cases (except VNF)
08:18:54 <jose_lausuch> not for now
08:18:58 <jose_lausuch> anyone else?
08:19:01 <morgan_orange> It is very early stage, but one of my colleague may add a test around energy audit
08:19:13 <jose_lausuch> energy?:)
08:19:36 <morgan_orange> through IPMI, it is possible to get info on power consumption, as we are able to automate deployment, tests,..it shall be possible to get a power footprint
08:19:55 <morgan_orange> and see from a stat perspective the delta due to teh creation of resources
08:20:11 <morgan_orange> still very early stage but if we do not try...we will never know...
08:20:20 <jose_lausuch> ok, that sounds original
08:20:43 <morgan_orange> if nothing to add on internal test cases, we can move to next topic
08:20:48 <morgan_orange> and a big one..
08:21:01 <jose_lausuch> i would never think about energy in this kind of things :)
08:21:05 <morgan_orange> #info early reflexion on test on power consuption POD audit
08:21:22 <morgan_orange> at the end it is a lot of $$$ for poor Telcos...
08:21:38 <morgan_orange> #topic Danube: Discussion on refactoring/templating
08:21:39 <jose_lausuch> like Orange? :)
08:21:57 <morgan_orange> in the Functest Danube page, there are already lots of items on the refactoring
08:22:01 <morgan_orange> of the framework
08:22:09 <morgan_orange> there are consideration on frgmenting docker
08:22:10 <jose_lausuch> come on, you have a lot of subscribers, I'm using orange at home :p
08:22:18 <morgan_orange> on creating more APIs
08:22:28 <jose_lausuch> I have some proposals about that
08:22:34 <morgan_orange> supporting Python 3
08:22:42 <morgan_orange> remove wild exit, call to bash from python
08:22:45 <morgan_orange> ....
08:22:56 <morgan_orange> so lots of things...
08:23:03 <morgan_orange> regarding docker
08:23:05 <jose_lausuch> yes, we need to split the task
08:23:14 <jose_lausuch> see the resources we have for Colorado
08:23:29 <jose_lausuch> call for volunteers for each task
08:23:31 <jose_lausuch> etc..
08:24:06 <morgan_orange> yep
08:24:24 <morgan_orange> I have some question on new API
08:24:43 <morgan_orange> for the moment we use a yaml file to declare the case
08:24:51 <morgan_orange> we have also such declaration in the DB
08:25:02 <Cristina__> -
08:25:05 <morgan_orange> so we duplicate the info
08:25:09 <jose_lausuch> yes
08:25:18 <morgan_orange> the CLI is based on the static file
08:25:21 <jose_lausuch> we could merge everything into one
08:25:31 <morgan_orange> but offers some feature that an APi could also offer
08:25:46 <morgan_orange> In the section https://wiki.opnfv.org/display/functest/Functest+Danube#FunctestDanube-Featureprojecttemplating/NewAPI
08:26:05 <morgan_orange> I tried to summarize the interaction a scenario owner and/or a feature project leader may have with Functest
08:26:16 <morgan_orange> if we consider the scenario owner
08:26:42 <morgan_orange> we laso got the request => he/she wants to know the runnable tests, be able to exclude some tests (he/she is master on board)
08:26:54 <morgan_orange> shall we add a method scenario to our API?
08:27:12 <morgan_orange> and complete the existing table to reflect the yaml file
08:27:24 <morgan_orange> the static file would be used only in backup if no connection is available
08:27:45 <jose_lausuch> you mean the scenario owner can use the api to enable/disable the tests he/she wants?
08:28:08 <jose_lausuch> we could generate the static file when building the docker image
08:28:14 <jose_lausuch> from the api
08:28:28 <morgan_orange> yes, through the API he will be able to create a list that will be stored in the DB. If nothing we use the default (current) behaviour) if not we use the custom list
08:28:49 <morgan_orange> jose_lausuch: yes something like that
08:29:22 <jose_lausuch> I would still use trhe static file inside functes cli, but generated dynamically from the api info
08:29:30 <morgan_orange> and we create a table scenario with info such as the owner, the custom list, the installer supported, the stats (what we have today in a csv file done for reporting)
08:29:33 <jose_lausuch> no need to do it real time
08:30:31 <morgan_orange> it is funny to see that we moved to scenario centric view, but we do not have scenario tables...
08:31:40 <morgan_orange> I will try to summarize
08:31:59 <morgan_orange> #info discussion on a possible evolution of the API with the creation of a scenario table
08:32:28 <morgan_orange> #info this table would include the scenario owner, the customTestlist (if tests excluded by the owner / default behavior), the score, the installer supported
08:32:52 <morgan_orange> #info the scenario owner could generate a custom list that will generate a yaml file used by the CLI and inside the container
08:33:07 <morgan_orange> #info if nothing customized we keep the current behavior
08:33:39 <morgan_orange> I can try to review a little bit teh proposable in the wiki page to reflect that if you are OK
08:33:59 <morgan_orange> #action morgan_orange refactor Danube page to reflect the introduction of a scenario table in the API
08:34:25 <morgan_orange> the second type of interaction is the feature test project versus Functest
08:34:29 <jose_lausuch> ok
08:34:31 <morgan_orange> I think we need more abstraction
08:34:34 <jose_lausuch> sound good
08:35:08 <jose_lausuch> any ideas how to achieve that?
08:35:13 <morgan_orange> a feature project has to: declare its project, provide the URL of the tests, reports results,..
08:35:24 <jose_lausuch> url of the tests?
08:35:37 <morgan_orange> how to run the tests in their repo
08:35:52 <jose_lausuch> ah ok
08:35:55 <morgan_orange> we could create a class
08:36:26 <morgan_orange> so feature projects that would like to use Functest to join CI/reporting/... would be more guided
08:36:36 <morgan_orange> ...
08:36:57 <jose_lausuch> we always try to make it easier for feature projects
08:36:58 <morgan_orange> I tried to list all the interactions I imagine from a feature project
08:37:06 <jose_lausuch> but always with wikis and presentations
08:37:22 <morgan_orange> - create my project in functest
08:37:26 <jose_lausuch> I hope we dont make it even more complex by doing this
08:37:40 <jose_lausuch> I would imagine a guided dashboard
08:37:46 <morgan_orange> -  use tools
08:37:57 <morgan_orange> - get/push logs
08:37:59 <jose_lausuch> where the scenario owner presses a bunch of buttons
08:38:02 <morgan_orange> - get/push results
08:38:13 <morgan_orange> - exclude from reporting
08:38:17 <morgan_orange> * add to dashboard
08:38:48 <morgan_orange> but you are right...is my view something that will suimplify or complexify...
08:39:34 <jose_lausuch> it could be a simple dashboard which converts scenario owners desires into calls to api and other rtyhings
08:39:42 <jose_lausuch> everything automated
08:39:49 <jose_lausuch> is that tto ideal? :)
08:40:35 <morgan_orange> maybe we should ask "old" feature project PTL to get their feedback
08:40:54 <jose_lausuch> sure
08:40:59 <morgan_orange> creating a class is a way for me for preformat and formalize the integration steps
08:41:09 <jose_lausuch> bryan sullivan would be happy to provide feedback I imagine :)
08:41:09 <morgan_orange> it is not very different from what we have
08:41:32 <morgan_orange> in Colorado => declaration in the DB + in the yaml file + run_tests + feature.py under testcases
08:42:16 <jose_lausuch> yes
08:42:21 <jose_lausuch> its a good idea
08:42:53 <morgan_orange> #action morgan_orange ask feature projects already integrated in Functest for feedback on a possible evolution considering Feature project as an object
08:43:21 <morgan_orange> the last API I see are teh cross test project APIs to avoid overlap
08:43:36 <morgan_orange> deploy a VNF, generate traffic, ...
08:44:09 <morgan_orange> as we in theory are able to deploy a vIMS thorugh the CLI, it would be interesting to offer this function to third party..
08:44:18 <morgan_orange> through an API
08:44:57 <morgan_orange> more complex but would give clarity on test domains...
08:45:18 <jose_lausuch> we can follow this up during the summit
08:45:24 <morgan_orange> anything help you want to add on the framework refactoring
08:45:29 <morgan_orange> oops
08:45:36 <morgan_orange> anything else (lapsus)
08:46:03 <morgan_orange> #topic Danube: Discussion on exception class
08:46:21 <morgan_orange> my idea was to discuss with other test projects to define an OPNFV exception class
08:46:42 <jose_lausuch> can you describe the idea behind?
08:46:50 <morgan_orange> yep
08:47:22 <morgan_orange> just to agree common error cause such as SUT not reachable, CI run failed, test DB not reachable, Feature test failed, ....
08:47:54 <morgan_orange> then instead of exit -1, we would raise exceptions (the same for all the test projects with possibility to customize some if needed)
08:48:03 <morgan_orange> and would have a consistent management on Jenkins
08:48:10 <jose_lausuch> ok, thanks
08:48:19 <morgan_orange> not difficult
08:48:22 <jose_lausuch> yes, it would make sense to define in releng
08:48:33 <morgan_orange> yes, palce is in relengneed an agreement of the
08:48:37 <morgan_orange> oops
08:48:40 <morgan_orange> yes in releng
08:48:48 <morgan_orange> need a agreement of all the test projects
08:48:57 <morgan_orange> I will suggest it next thursday
08:49:02 <jose_lausuch> ok
08:49:12 <jose_lausuch> something for all test projects
08:49:26 <jose_lausuch> for common potential errors
08:49:30 <morgan_orange> #info for all the test project, list possible error causes and create a class in Releng
08:49:30 <jose_lausuch> good idea
08:49:47 <morgan_orange> #info could behelpful to have a consistent way to manage error at the test level and avoid wild exit
08:49:53 <morgan_orange> #topic Meetup
08:50:14 <morgan_orange> as José coudl be our guide in Barcelona...the Lannion meetup is not more needed
08:50:23 <morgan_orange> #info meetup in Lannion canceled
08:50:41 <morgan_orange> #info proposal to organize the meetup in Barcelona Tuesday 25th
08:50:47 <morgan_orange> #info room has been asked
08:51:01 <morgan_orange> of course people may have to attend presentations, booth,...
08:51:14 <jose_lausuch> dont say: not needed, better say : postponeed (still looking forward to seeing brittany)
08:51:35 <morgan_orange> but juhak, Serena, Jose, Cedric, Morgan, Luke,, May, maybe Valentin will be there....
08:51:57 <morgan_orange> #info people who are free can also attend OPNFV meetup in Paris on the 21th
08:52:27 <morgan_orange> #info Lannion candidate for a future meetup...
08:52:36 <morgan_orange> #topic AoB
08:52:42 <jose_lausuch> I am working on something in releng that might be useful for us
08:52:48 <morgan_orange> #info update o the reporting with d3js for gauge and trend line
08:52:54 <morgan_orange> #info export to pdf in progress but js svg not displayed
08:53:00 <morgan_orange> #info export to csv also planned
08:53:07 <jose_lausuch> it's an installer adapter, you'll see this week a big patch
08:53:18 <morgan_orange> #link http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
08:53:28 <morgan_orange> #info installer adapter in the pipe....
08:53:45 <morgan_orange> it will somehow abstract the installer to the rest of the world?
08:54:06 <morgan_orange> #info discussion on CI evolution with releng project
08:54:25 <morgan_orange> #link https://wiki.opnfv.org/display/INF/CI+Evolution
08:54:25 <jose_lausuch> something like that
08:54:32 <morgan_orange> #action all review CI evolution
08:54:38 <jose_lausuch> so, you can get the ips of thenodes
08:55:00 <jose_lausuch> and scp to them or run remote commands doing ssh-proxying the installer
08:55:04 <jose_lausuch> and so on
08:55:26 <morgan_orange> ok, it corresponds to the need we mentoned  to be able to connect to nodes of the SUT for feature people
08:55:35 <jose_lausuch> yes
08:56:09 <jose_lausuch> no ugly scrip fetch_os_creds.sh with looong sed grep commnds
08:56:14 <morgan_orange> ok let's wait and see but it will be great to have something methods like connectControlNode without divin into each installer
08:56:17 <jose_lausuch> everything python based :)
08:56:57 <morgan_orange> for CI evoolution, I think adapations in functest are not so big...even if there was a misunderstanding
08:57:05 <jose_lausuch> yes
08:57:07 <morgan_orange> we considered weekly loop for long duration tests,
08:57:18 <morgan_orange> in CI it is more considered as propotion of daily
08:57:34 <morgan_orange> so it is a bit different but we can converge
08:57:41 <morgan_orange> I will had this topic for next week
08:57:46 <morgan_orange> as weel as a status on VNF
08:57:55 <morgan_orange> but if you want to add a topic, do not hesitate
08:57:55 <jose_lausuch> maybe we could also discuss it in the infra meeting
08:58:29 <morgan_orange> yep
08:58:29 <morgan_orange> I think the proposal of Fatih is clear, we have to adapt to it
08:58:56 <morgan_orange> but resources will be needed and adapations on installer side..which is something that is requested for a long time...
08:59:14 <morgan_orange> but if we want to optimize CI, this promotion mechanism should be put in place
08:59:37 <morgan_orange> any other topic for the AoB for today
08:59:38 <morgan_orange> ?
08:59:53 <jose_lausuch> agree
09:00:12 <morgan_orange> hideyasu_ool:  Cristina__ ollivier Raghav ....
09:00:26 <Raghav> Ok
09:00:33 <Cristina__> agree
09:00:37 <hideyasu_ool> agree
09:00:56 <morgan_orange> #info CI evolution: mechanism described by Fatih clear, Functest could adapt.
09:01:08 <morgan_orange> #info ressources needed and adoption of the procedure by all the installers
09:01:11 <morgan_orange> ok
09:01:14 <morgan_orange> thanks everybody
09:01:20 <morgan_orange> Have a good week
09:01:25 <morgan_orange> see you next week
09:01:32 <jose_lausuch> have a good week
09:01:32 <morgan_orange> and do not hesitate to ask for topic slots
09:02:04 <morgan_orange> next week we will have a younger, fitter, smarter and nicer PTL...
09:02:06 <morgan_orange> :)
09:02:14 <morgan_orange> #endmeeting