08:00:01 #startmeeting Functest weekly meeting March 8th 08:00:01 Meeting started Tue Mar 8 08:00:01 2016 UTC. The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:01 Useful Commands: #action #agreed #help #info #idea #link #topic. 08:00:01 The meeting name has been set to 'functest_weekly_meeting_march_8th' 08:00:07 #info Morgan Richomme 08:00:39 #info Viktor Tikkanen 08:01:14 #info agenda https://wiki.opnfv.org/functest_meeting 08:01:22 hi viktor_nokia 08:01:36 Hi Morgan! 08:02:38 jose_lausuch: is celebrating brahmaputra ins Stockholm 08:02:48 :) 08:03:07 BTW, juhak is on vacation this week... 08:03:12 so it should be quiete 08:04:03 let's discuss the different point of the agenda, Asian contributors may come and other people could see off line 08:04:10 #topic action points follow up 08:04:19 #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-03-01-08.00.html 08:04:36 #info May-meimei did you have a look at the OPNFV Beijing meet-up group? 08:05:15 #info clean of the committer in progress, add of new committer OK, removal is a bit longer (need the green light of TSC) but it shall be OK soon 08:05:43 #info mail to express concern on git management done => we do everything in Master, bug fixes reported to Stable 08:05:59 #topic SR1 08:06:17 #info quiet post Brahmaputra time regarding the release 08:06:33 #action morgan_orange check new scenario candidates 08:07:01 #info Release manager is no more working in Linux FOundation (mail received on Friday), self organization to be planned for SR1 08:07:22 #info CI still working but no significant change since B release 08:07:42 #topic focus on Tempest 08:08:05 I think it would be nice to have a new look on the results 08:08:17 #link http://testresults.opnfv.org/dashboard/ 08:09:18 BTW, I 've installed official Brahmaputra with Apex in our internal lab 08:09:19 #info apex/odl_l2 60-80%, apex/onos 75-92%, apex/odl_l3 56-68% 08:09:27 cool and whart are the results 08:09:38 I was wondering on the influence of the POD on the results 08:09:46 as Apex is the only installer when we have only 1 reference 08:10:30 tempest results were: 08:10:34 Ran: 209 tests in 3310.0000 sec. - Passed: 200 - Skipped: 3 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 6 Sum of execute time for each test: 2801.4488 sec. 08:11:00 I had to use --serial option because we have only few floating IPs 08:11:22 OK 08:11:23 hi 08:11:29 #info jose_lausuch 08:11:32 partially 08:11:34 so it shows the influence of the POD... 08:12:02 sitting next to fdegir :) 08:12:10 the LF POD1 seems slower compared to Ericsson POD2, Orange POD2, ... 08:12:21 I run the tests from docker image installed on jumphost (if this makes some sense) 08:12:23 hi jose_lausuch, king regards to MC Fatih 08:12:59 anac1 is also here :) 08:13:40 viktor_nokia: shall we move officialy to --serial ? 08:13:46 viktor_nokia: I think we can communicate towards apex installer/ releng on that, we have somehow the same issue than with Intel/Orange pod 2 pod...we know that results of same tests with same installer lead to different results that may be really misleadign for a global evaluation of the system 08:13:51 what is the time difference? 08:14:08 As far as i remember we moved to --serial 08:14:19 on all the labs, did'nt we 08:14:34 There are few test cases that fail with timeout error (and timeout is 300s) 08:14:44 This impacts the total time 08:14:57 especially in case of serial execution 08:15:04 morgan_orange: no, cmd_line = "rally verify start " + OPTION + " --system-wide" 08:15:13 jose_lausuch: ok 08:15:21 I believed we did so 08:15:32 that is something we can try and merge and see the influence on LF POD1 08:15:43 yes 08:15:54 jose_lausuch: Say Olá to Ana as well 08:16:03 ok :) 08:16:26 #action viktor_nokia set --serial options in repo to see influence on LF POD1 08:16:28 jonas bjurel and Chris also here 08:16:33 the dream team 08:16:39 we can try for a while the --serial 08:16:44 and see the results 08:17:07 yes, I set the action point to viktor_nokia I will merge and cherry pick on Stable 08:17:19 ok 08:17:51 It will be interesting to get a predeployement tool to get an audit of the POD, I already evoked that with qtip people 08:18:20 but it is difficult to justify strict success criteria if we are able to show very different results from one POD to another.. 08:19:22 #action viktor_nokia send a mail to apex/pharos to indicate tempex results on apex/Nokia and show the different with results got on LF POD1 08:19:47 #info compass/nosdn 100%, compass/odl-l2 94-100%, compass/onos 94%, compass/ocl 31% 08:19:51 ok 08:21:42 #info fuel/nosdn 86-98% (better results on LF POD2 than Ericsson POD2), fuel/odl_l2 88-95%, fuel/onos 91-95%, fuel /odl_l3 79-92%, fuel/bgpvpn 76%; fuel/nosdn_ovs 98-100% 08:22:06 btw, did my patch work? to have rally and tempest preinstalled? I havent had time to check jenkins 08:22:06 it is strange to have difference between odl_nosdn and odl_nosd-ovs 08:22:21 jose_lausuch: I did not check as well 08:22:46 seems that last run still OK.. 08:24:06 but if I compare last compass run with a previous one....last try lasts 2h39' and some days ago, it took 1h55' 08:24:14 not sure if it is due to the change 08:24:28 to be checked 08:24:46 it strange, we should save some time by having it pre installed.... 08:24:48 yes 08:24:53 can you action me? 08:25:11 I will do a change also to have vIMS and Promise dependencies pre installed in the docker image 08:25:23 #info joid/nosdn ((Orange POD2) 88%, joid/odl_l2 98%, joid/onos: 87% 08:25:26 the functest installation take to long when installing tempest and promise 08:25:56 it is probably a topic for discussion in Espoo 08:26:29 shall feature projects be always integrated in Functest docker image or shall we have something more modular 08:27:41 viktor_nokia: coudl we have a summary of the errors per pod/scenario, I think that most of them are common and we can have some indication of the influence on the scenario on the tempest results 08:27:59 I do not know for the future if we can keep the same custom list whatever the scenario 08:28:33 I can help doing that 08:28:46 #action morgan_orange initiate a page on Tempest errors per scenario/installer 08:28:57 I can make some summary, will take some time... 08:29:00 #action viktor_nokia reference apex tempest errors 08:29:09 #action jose_lausuch reference fuel tempest errors 08:29:19 #action morgan_orange reference joid errors 08:29:19 shall we have a common format? 08:29:31 #action May-meimei reference compass tempest errors 08:30:04 jose_lausuch: viktor_nokia I suggest I initiated the page, and once done we validate the format 08:30:08 ok 08:30:15 we probably could also automate a little bit such things.. 08:30:35 we can parse the logs from artifacts 08:30:35 I have still the action plan to automate the weather forecasts...it is a good transition to the other topic 08:30:39 and build something automatic 08:30:48 better than maintaining a wiki 08:30:59 jose_lausuch: yes it will be better and we can reuse Valentin's framework 08:31:14 it is in python and able to call APi to dynamically create the page 08:31:23 so let's try to do it thsi way 08:31:34 we could decide to stop doing things an unmaitainable wiki... :) 08:33:13 #link http://testresults.opnfv.org/reporting/vims/ 08:33:36 I agree 08:33:54 the code is here https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/tools/reporting 08:34:05 we could have something similar for tempest 08:34:11 and for the weather forecast 08:34:36 yep 08:34:48 #info tempest and general status to be reported dynamically (decision to avoid using wiki for such things as it is long and difficult to maintain) 08:35:27 #topic success criteria 08:35:45 #info during last test weekly meeting we initiated a discussion on the success criteria 08:36:04 #info there is also a mail thread => shall criteria be more strict 08:36:27 as indicated assuming that we can clearly see a big dependency towards the POD it is not easy to justify... 08:36:58 Moreover criteria for performance tests are obvious, for functional tests it is a bit test by test 08:37:18 see for Brahmaputra 90%, vPing, vIMS, ...we almost decided all the criteria 08:37:24 it is also a topic for Espoo 08:37:35 how to define such criteria 08:37:56 #info discussion on test criteria to be done during the Meet-up 08:38:49 #action morgan_orange share first version of Meet-up agenda before the end of the week 08:38:58 I would like also to add consideration on security 08:39:11 viktor_nokia: I saw that there is someone from Nokia dealing with opnfv-security 08:39:18 do you know him? 08:39:38 hmmm... I need to check... 08:39:42 I think it would make sense to have a project dealing with test security 08:39:57 hardening of VMs, ... 08:40:52 David_Orange: should start working on security groups (which is already a first step) but when discussing with operational teams, a security audit of the system is missing 08:41:16 viktor_nokia: the guy is Luke Hinds 08:41:23 #link https://wiki.opnfv.org/security 08:41:43 yes, he is involved into OPNFV Security Group but I don't know him personally 08:42:16 Ok I can contact him to see if they planned to implement tests, as far as I understand for the moment it is more guides 08:42:53 #action morgan_orange contact OPNFV security group to see if test projects related to security are planned or not for C release (maybe a topic also for the Meet-up) 08:43:09 #topic API/Dashbopard evolution 08:43:31 #info I got some question from Cisco people on the APi, they have good idea of evolution of the API 08:44:30 #info and they are working on ELK (Elastic Search LogStash Kibana), they are currently exporting mongo info to use them in Kibana and try to reproduce what we have in our basic dashboard (based on MIT lib dygraph) 08:45:05 #info evolution of the API and the dashboards to be discussed in the Meetup 08:45:18 I asked whether they will contribute to Functest (as peter left) 08:45:30 wait for thei answer, I invited them to join the Meetup 08:45:41 we should have also a new E/// contributor soon 08:46:40 Nikolas who worked on bgpvpn project and has high skill in Rally/tempest/... shall join, we could organize a more formal introduction next week. I would like to invite the new contributor and let them the floor (so need GTM) to express their view with a new look 08:46:55 #action morgan_orange organize GTM next week for weekly meeting, keep a slot for new contributors 08:48:13 #info on the APi, we should add new fields (version/scenario) and a suite of unit tests is needed to ensure the stability of the APIs (migration or at least data change will be needed), option for the time search, for now we can use period=N to retrieve the last N days, but there is not API way to retrieve the last 5 scenarios X of installer Y 08:48:59 #info that is whty for the automation of the forecast => only the results of the last N days (throuhj period param) can be retrieved and then a post processing is needed => connected to teh discussion on success criteria 08:49:10 #topic task allocation 08:49:44 #info we will wait the introduction of new contributor but we shall be able to allocate the main tasks as soon as we know them 08:50:01 #info roadmpat for C release is expected after the meet up 08:50:17 but we need more feedback from our Asian contributors 08:50:31 #topic AoB 08:50:42 any information you want to share? 08:51:06 not from my side... 08:51:25 yes, https://www.youtube.com/watch?v=rOwN3m0045c 08:51:34 for the CFP berlin viktor_nokia would it make sense to propsoe something dealing with troubelshooting Tempest accross the different PODs, a challenge to ensure stability and interoperability 08:51:41 I will create a new version soon speaking myself, and upload it to opnfv channel :) 08:51:54 cool 08:52:34 #info Jose Lausuch international Entertainement soon presents Functest versus Marvels 08:52:40 #link https://www.youtube.com/watch?v=rOwN3m0045c 08:52:58 :) 08:53:31 viktor_nokia: for teh CFP for berlin, we should submit before End of march, so if you are interested I think we can draft something involving you, Jose and myself 08:54:01 ok for me 08:54:07 what is CFP? tried google already... 08:54:13 Call For Paper 08:54:22 we can submit presentation for the Summit 08:54:27 viktor_nokia: to submit the presentation ideas for the submit 08:54:35 I already exchanged with jose on functest in general 08:54:37 summit* 08:55:03 In principle OK for me but I'm not 100% sure if I will be there 08:55:12 but something dedicated to tempest and how representative it is, explaining also the influenc of the POD,..;I think it could be useful to the community 08:55:30 yes, definitely 08:56:07 #action morgan_orange initiate Tempest presentation on Berlin 08:56:20 anything else you want to share? 08:56:30 jose_lausuch: I have a meeting with .... E// in 4 minutes 08:56:37 nope 08:56:41 morgan_orange: cool 08:56:42 Nokia public lab is almost ready 08:56:53 let's info that 08:57:01 #info Nokia public lab soon ready 08:57:07 it will be based on apex, I assume 08:57:12 yes 08:57:21 great to have a second source.. 08:57:22 we have Brahmaputra there currently 08:57:52 ok thanks all, see you next week, hope I will be able to deal with all my action points....lots of internal communication ahead to explain what we have done :) 08:57:58 #endmeeting