08:01:23 <morgan_orange> #startmeeting Functest weekly meeting May 31th
08:01:23 <collabot`> Meeting started Tue May 31 08:01:23 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:01:23 <collabot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:01:23 <collabot`> The meeting name has been set to 'functest_weekly_meeting_may_31th'
08:01:33 <morgan_orange> JuhaHaapa: need more coffee
08:01:40 <morgan_orange> #topic call roll
08:01:40 <lhinds> #info lhinds
08:01:44 <morgan_orange> #info Morgan Richomme
08:01:48 <JuhaHaapa> #info Juha Haapavirta
08:01:49 <juhak> #info Juha Kosonen
08:01:52 <viktor_nokia> #info Viktor Tikkanen
08:01:52 <morgan_orange> #info agenda: https://wiki.opnfv.org/display/functest/Functest+Meeting
08:01:57 <jose_lausuch> #info Jose Lausuch
08:01:57 <morgan_orange> #topic action points follow-up
08:02:01 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-05-24-08.00.html
08:02:04 <JuhaHaapa> morgan_orange: :):)
08:02:18 <morgan_orange> #info AP1: refactoring Testcase.yaml done
08:02:33 <morgan_orange> #info AP2: group Rally full and Tempest full in 1 Tier done
08:02:40 <jose_lausuch> yes
08:02:46 <morgan_orange> we may have a discussions on Tier / yardstick and common definition
08:03:01 <morgan_orange> #info AP3 integration of success criteria directly in testcases.yaml
08:03:18 <morgan_orange> #info rather than hardcoded in the code of the test cases
08:03:27 <morgan_orange> I think we need also to discuss this a little bit
08:03:45 <jose_lausuch> this is not done yet :)
08:03:47 <morgan_orange> as success criteria (condition to set the criteria to passed or failed in the DB)
08:03:58 <morgan_orange> corresdpond to variables
08:04:07 <morgan_orange> so if we put a condition in the yaml file
08:04:14 <morgan_orange> the var shall correspond...
08:04:21 <morgan_orange> e.g. duration < 1800
08:04:33 <morgan_orange> we supposed that we use duration as a variable name
08:04:53 <morgan_orange> I imagine we need to list all the var we use in order to set up the criteria
08:05:02 <morgan_orange> and try to be consistant
08:05:33 <morgan_orange> #info AP4 add security_scan in DB and in testcase.yaml done
08:05:52 <morgan_orange> we will see later, the CI pipeline integration has to be done, but almost everything should be in place
08:06:10 <jose_lausuch> I can do a proposal in the yaml
08:06:35 <morgan_orange> #info AP5 add start/stop fields in API: done on refactored API (not in testresults.opnfv.org yet) but hopefully before the ned of the week (Serena did a great job)
08:06:39 <morgan_orange> jose_lausuch: ok
08:07:00 <morgan_orange> #info AP6 add filter param for API: not done yet
08:07:17 <morgan_orange> #info AP7: add type, also ready for next version of the API
08:07:27 <morgan_orange> and that is all for the APs
08:07:42 <morgan_orange> #topic Sprint #8 follow-up
08:07:53 <morgan_orange> before diving into JIRAs
08:08:03 <morgan_orange> #info First run on Master for Apex
08:08:27 <morgan_orange> #info Functest jenkins dashboard refactored (thanks to May-meimei)
08:08:36 <morgan_orange> #link https://build.opnfv.org/ci/view/functest/
08:09:10 <jose_lausuch> and we have job mismatching
08:09:13 <morgan_orange> apex is not in this page ...
08:09:16 <jose_lausuch> its apex build daily
08:09:32 <May-meimei> morgan_orange: you are welcome
08:09:47 <May-meimei> yes,we may miss some jobs
08:10:45 <morgan_orange> last apex success https://build.opnfv.org/ci/job/functest-apex-apex-daily-master-daily-master/
08:11:12 * fdegir will hopefully send patches to align apex jobs
08:11:15 <jose_lausuch> we still have the problem of duplicated output, even with CI_DEBUG=false
08:11:16 <jose_lausuch> so bad...
08:11:25 <jose_lausuch> it's cinder python module's fault...
08:11:28 <morgan_orange> probably some troubleshooting ahead (ODL, tempest,...)
08:11:33 <jose_lausuch> fdegir: thanks
08:11:57 <morgan_orange> fdegir: thanks, if you succeeded in aligning Apex CI you definitively deserved the yearly award... :)
08:12:18 <fdegir> morgan_orange: we'll see
08:12:31 <morgan_orange> #action morgan_orange jose_lausuch viktor_nokia juhak have a look at first Apex Master logs
08:12:59 <jose_lausuch> 2016-05-30 17:21:54,189 - healtcheck - INFO -  Health check passed
08:13:12 <morgan_orange> #info fist compass/ocl also integrated
08:13:13 <jose_lausuch> 2016-05-30 17:22:54,970 - vping_ssh - INFO - vPing detected!
08:13:23 <jose_lausuch> 2016-05-30 17:23:40,317 - vping_userdata - INFO - vPing OK
08:13:42 <jose_lausuch> 2016-05-30 17:35:06,471 - run_tempest - INFO - Results: {'timestart': '2016-05-3017:24:07.434810', 'duration': 654, 'tests': 169, 'errors': '{0}
08:13:55 <viktor_nokia> vPing duration:'25.6' s  !
08:14:01 <jose_lausuch> what????
08:14:04 <jose_lausuch> ah vping
08:14:06 <jose_lausuch> :)
08:14:18 <jose_lausuch> tempest took 11 min
08:14:30 <jose_lausuch> and 169 tests... I thought smoke had less
08:14:32 <jose_lausuch> around 100 or so
08:14:44 <morgan_orange> functesting overall last 40 minutes
08:14:45 <jose_lausuch> but 169 tests in 11 minutes in serial is a record!
08:14:57 <morgan_orange> which is what we announced...
08:15:06 <morgan_orange> we can go to vacations :)
08:15:22 <morgan_orange> or strike (French habit at the moment)
08:15:24 <jose_lausuch> 95% rally
08:15:27 <jose_lausuch> haha
08:15:52 <jose_lausuch> odl tests failed...
08:16:20 <viktor_nokia> is it baremetal or virtual deployment?
08:16:25 <morgan_orange> baremetal
08:16:32 <morgan_orange> lf pod1 (you have access now...)
08:16:45 <morgan_orange> doctor is ok
08:17:11 <morgan_orange> security_scan also :) but we did not do anything but from CI, we are ready
08:17:39 <morgan_orange> and that is all as vIMS, full Tempest full Rally are on weekly loop
08:17:45 <morgan_orange> so good results
08:18:06 <morgan_orange> #action ollivier look at ODL results on Apex/Master scenario
08:18:23 <morgan_orange> #action juhak see the reasons of the 95% on rally
08:18:30 <juhak> about failed rally cases,
08:18:37 <juhak> heat: https://jira.opnfv.org/browse/APEX-161
08:18:44 <juhak> nova: https://jira.opnfv.org/browse/APEX-147
08:18:47 <jose_lausuch> wow
08:18:48 <jose_lausuch> :)
08:18:56 <jose_lausuch> the AP is already done
08:18:57 <morgan_orange> ok you AP is already closed then
08:19:04 <jose_lausuch> juhak: that's efficiency! :D
08:19:10 <juhak> :)
08:20:00 <morgan_orange> ok anyway good results for a first try
08:20:33 <morgan_orange> there was a discussion on the timeout, the apex tests include the timeout for the openflow creation rules?
08:21:10 <jose_lausuch> in the healthcheck? I think so
08:21:14 <jose_lausuch> its for every installer
08:21:28 <viktor_nokia> I'm trying to reproduce the issue in our lab with the latest Apex but deployment fails...
08:21:40 <morgan_orange> sometimes it happens...
08:22:14 <viktor_nokia> anyway, it is in https://jira.opnfv.org/browse/APEX-149
08:22:15 <morgan_orange> the timeout was needed mainly for Apex, right?
08:22:28 <viktor_nokia> not 100%
08:22:28 <jose_lausuch> yes
08:22:31 <morgan_orange> #info Rally errors analyzed, 2 jora created
08:22:43 <jose_lausuch> but maybe a sleep 10 is enough, but to be safe we added ssleep 60
08:22:51 <morgan_orange> #link https://jira.opnfv.org/browse/APEX-161
08:22:58 <viktor_nokia> because  e.g. vPing duration is quite high also on some other installers
08:22:58 <morgan_orange> #link https://jira.opnfv.org/browse/APEX-147
08:23:05 <jose_lausuch> vping doesn't use sleeps, just the time it needs to process the code and spawn the vms
08:23:11 <jose_lausuch> since healthcheck is bash, it might be faster
08:23:15 <viktor_nokia> only Compass (and now Apex) are OK
08:24:00 <morgan_orange> for joid we have this run https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-master/lastSuccessfulBuild/consoleText
08:24:23 <morgan_orange> vPing SSH KO
08:24:35 <morgan_orange> vPing userdata OK
08:24:52 <morgan_orange> Tempest Results: {'timestart': '2016-05-2611:28:40.882648', 'duration': 2710, 'tests': 169, 'failures': 18}
08:25:06 <viktor_nokia> vping_ssh duration should be max 30-40 sec (response to the first DHCP request)
08:25:19 <morgan_orange> Rally 100%
08:25:42 <morgan_orange> Promie 29/33
08:26:07 <morgan_orange> it seems that we have some open issue to push results to artifacts
08:26:44 <morgan_orange> on Intel POD6 gsutil not installed
08:27:00 <morgan_orange> #action May-meimei have a look why gsutils not installed on CI POD Intel POD6
08:27:26 <morgan_orange> we have also tests with fuel, but as far as I know Master = Brahmaputra for the moment
08:27:29 <jose_lausuch> its just on that pod
08:27:30 <jose_lausuch> ?
08:27:33 <morgan_orange> I have also some doubt on joid
08:27:50 <morgan_orange> jose_lausuch: uploading on lf-pod1 seems OK
08:28:40 <May-meimei> morgan_orange: sorry, I can't access to intel lab
08:29:06 <morgan_orange> May-meimei: OK can you see with jmorgan1?
08:29:18 <morgan_orange> Back on Sprint 8
08:29:25 <morgan_orange> #link https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=59&view=planning.nodetail
08:29:52 <morgan_orange> any concerns? as you probably know "feature" freeze is now planned for beginning of July
08:30:00 <morgan_orange> it includes our internal features
08:30:08 <morgan_orange> for integration we will have more time
08:30:15 <jose_lausuch> jmorgan1 is the lab owner for intel, he should be able to fix it
08:30:42 <morgan_orange> BTW OVNO test suites will be rewritten (src Stuart Mackie) so I comment OVNO script
08:31:10 <morgan_orange> we also initiate a discussion with Yardstick on where feature shall be tested
08:31:40 <morgan_orange> it seems that some feature projects created functional tests but their project is asosciated with Yardstick
08:31:57 <morgan_orange> If we consider that both projects are umbrella projects (for Functionnal and Performance testinà
08:32:11 <morgan_orange> it coudl make sense to re import some test cases in Functest to be more consistant
08:32:24 <morgan_orange> discussion initiated yesterday, wait for kubi feedback
08:32:34 <morgan_orange> the idea is not to do everything in Functest...
08:32:47 <morgan_orange> just to be clear for the feature projects and the end users
08:33:29 <morgan_orange> and regarding the JIRA
08:33:37 <morgan_orange> everybody is comfortable with the timing?
08:33:43 <morgan_orange> no help needed?
08:33:46 <viktor_nokia> any progress with "offline functest"? do we have any images (e.g. cirros) included into functest container?
08:33:50 <jose_lausuch> I might have a concern
08:34:10 <morgan_orange> viktor_nokia: the main issue will be probably vIMS
08:34:26 <morgan_orange> valentin is coming back next week, but there may be some issues with this testcase
08:34:34 <jose_lausuch> I dont have time for this https://jira.opnfv.org/browse/FUNCTEST-157
08:34:41 <jose_lausuch> that's a uge change
08:34:46 <jose_lausuch> maybe for D-river
08:35:17 <jose_lausuch> viktor_nokia: we do have a cirros in the container
08:35:19 <morgan_orange> jose_lausuch: yep it seems more reasonable (you already made a first big refactoring)
08:36:10 <viktor_nokia> jose_lausuch: OK, then I can probably proceed with https://jira.opnfv.org/browse/FUNCTEST-202
08:36:10 <morgan_orange> any objection to move this JIRA from C to D
08:36:26 <jose_lausuch> I hope not :)
08:36:46 <jose_lausuch> viktor_nokia: ok, I think the main reason for having internet is because of some rally stuff
08:36:56 <jose_lausuch> ah
08:36:59 <jose_lausuch> and for vIMS!
08:37:12 <jose_lausuch> but maybe since vIMS is not running daily...
08:37:16 <morgan_orange> but vIMS is now in VNF Tier...
08:37:27 <morgan_orange> it is a nice to have :)
08:37:28 <juhak> I'll change those rally cases that currently requires internet
08:37:35 <viktor_nokia> shoul we re-decide then if "offline" functionality is needed?
08:37:37 <jose_lausuch> difficult to test offline functest on ci pods where internet is available :)
08:37:46 <jose_lausuch> I would say is needed
08:37:50 <jose_lausuch> at least for the basic tests
08:37:53 <morgan_orange> people from cloudify and metaswitch will be in berlin, as well as Valentin, we may imagine some progress there
08:38:09 <jose_lausuch> juhak:  what test cases ?
08:38:37 <juhak> two cases on 'requests' set
08:38:40 <jose_lausuch> can you write the needed changes in the jira issue as comments?
08:38:41 <jose_lausuch> ok
08:39:01 <juhak> ok, I'll update jira
08:39:25 <JuhaHaapa> for me working behind the proxy is a killer. Colum and me are working our way through that jungle, and hopefully making some more notes in the config guide
08:39:32 <viktor_nokia> we need to disable internet (globally) for a couple of hours to test the feature :)
08:40:15 <morgan_orange> ok so JIRA seems under control
08:40:17 <jose_lausuch> JuhaHaapa: ok.. I don't undestand how companies are trying to become open source without giving the employees the proper tools.... :)
08:40:22 <morgan_orange> probably doc still a bit understimated
08:40:38 <morgan_orange> even if with CLI, almost no need is needed now :)
08:41:03 <morgan_orange> #topic Berlin Summit
08:41:09 <jose_lausuch> morgan_orange: for the user guide, we can skip the scripts call, just using the cli should be enough for the user
08:41:18 <morgan_orange> jose_lausuch: +1000
08:41:28 <jose_lausuch> the user doesnt need to understand what scripts do what, that's why we have CLI
08:41:29 <JuhaHaapa> jose_lausuch: Yes exactly :)
08:41:31 <morgan_orange> and we should also convince to stop doing rst for tests, noone reads that
08:41:37 <jose_lausuch> JuhaHaapa: so, less work :)
08:41:38 <morgan_orange> for test results
08:41:59 <jose_lausuch> yes, we have your automatic reporting tool which is 1000 times better
08:42:11 <morgan_orange> for Berlin, I shared a presentation, just wait for Sofia feedback (opfnvdocs next meeting is planned for the 8th of June)
08:42:34 <jose_lausuch> I still need to prepare the presentation
08:42:41 <morgan_orange> so until I merge you cannot contribute...even if there are some parts where mmost of you could contribute
08:43:06 <morgan_orange> #info pres Berlin under review, contributions welcome
08:43:16 <morgan_orange> #info we will probably needs several breakout sessions
08:43:23 <jose_lausuch> you got a +2 , you can merge :)
08:43:24 <morgan_orange> #info one on D river already....
08:43:48 <morgan_orange> #info one for integration, several feature projects will be there, the design summit would be convenient to work on integration
08:44:23 <morgan_orange> #link https://wiki.opnfv.org/display/EVNT/Berlin+Design+Summit+Planning
08:44:37 <morgan_orange> programm of the design summit should be available soon
08:44:55 <morgan_orange> several Functest contributors proposed things (security, test APi, vIMS, ...)
08:45:24 <jose_lausuch> that's good
08:45:36 <jose_lausuch> lot of contributions from functest
08:45:39 <morgan_orange> after Functest slot (on Monday) there is also a session "Conversations with the oPNFV Test community"
08:45:43 <jose_lausuch> I dont think all of them will make it
08:45:51 <jose_lausuch> but at least we'll have some representation :)
08:46:05 <morgan_orange> the idea is to introduce all the test projects and try to give our vizion as a test community
08:46:15 <morgan_orange> work in progress during the Thursday weekly meeting
08:46:25 <morgan_orange> if you have any commentsinputs feel free
08:46:48 <morgan_orange> #topic Test conditions definition in testcase.yaml
08:46:51 <JuhaHaapa> my travel plan seems to get approval at the end of August :)
08:46:54 <morgan_orange> #info already mentioned in the introduction
08:47:25 <morgan_orange> #topic Functest test coverage Map
08:47:41 <morgan_orange> one of the discussion with the other test projects is the notion of test coverage
08:47:54 <jose_lausuch> kubi replied
08:48:06 <morgan_orange> for D river we shall be able to initiate something (more automated) on this topic
08:48:22 <morgan_orange> the idea is to identify the uncovered or poorly covered areas (and there are some...)
08:48:22 <jose_lausuch> we can bring this to the community,in the mailing list and we can say something on thursday (test WG)
08:48:36 <morgan_orange> jose_lausuch: OK cool
08:48:56 <morgan_orange> I will plan a slot for this then
08:49:05 <jose_lausuch> ok
08:49:22 <morgan_orange> #link https://wiki.opnfv.org/display/functest/Testing+Coverage
08:49:46 <morgan_orange> #info for testing coverage a long long time ago we suggested 3 representations Domain/Component/ETSI
08:50:19 <morgan_orange> #action all contribute to domain page
08:50:27 <morgan_orange> do not hesitate to suggest another model
08:50:53 <morgan_orange> #topic Functest / ETSI TST group
08:51:23 <morgan_orange> connected to previous topic and raised during the Plugfest: shall we try to formalize Functest and discuss this with ETSI
08:51:48 <morgan_orange> yardstick is implementing ETSI Testcase definition
08:51:58 <jose_lausuch> that's a high level administration task :)
08:52:12 <jose_lausuch> who else is in contact with ETSI besides you?
08:52:25 <morgan_orange> shall we propose Functest as a methodology/framework to validate a Telco Cloud...
08:52:27 <jose_lausuch> I think it's a good idea
08:52:36 <jose_lausuch> yes
08:52:40 <morgan_orange> not a bad idea but lots of admin work...as you said...
08:52:49 <jose_lausuch> When we get more mature, it could be an idea
08:52:58 <morgan_orange> I think we all have an ETSI correspondent...
08:53:06 <jose_lausuch> we need a sales man !
08:53:07 <morgan_orange> If you could send some unformal probes
08:53:13 <morgan_orange> to get feedback
08:53:57 <morgan_orange> I think we need probably more maturity but if we consider the plugfest, the discussion on certification...we have already something (better than nothing isn't it...)
08:54:16 <jose_lausuch> for sure
08:54:20 <morgan_orange> #action all contact ETSI colleague to get feedback on the opportunity to formalize Functest for ETSI /TST
08:54:37 <jose_lausuch> however I dont think this is very urgent
08:54:45 <jose_lausuch> but if we have time to start discussions...
08:55:05 <morgan_orange> now and I am not sure there will be lots of volunteer to reverse engineered our work into standards...
08:55:30 <morgan_orange> anyway, let's ask the question
08:55:35 <morgan_orange> #topic AoB
08:56:17 <morgan_orange> #info thanks to SerenaFeng, the API is almost fully  (and nicely) refactored => need to share it with the community in Berlin
08:56:37 <morgan_orange> #info target put it in testresults.opnfv.org before the end of the week
08:56:55 <lhinds> quick one from me, I will likely need some hand holding for when security_tests are merged and active in CI
08:57:24 <lhinds> mainly around how things work in jenkins, just until I get up to speed
08:57:31 <morgan_orange> lhinds: I will be there
08:57:40 <lhinds> thanks morgan_orange !
08:57:41 <morgan_orange> as you can see in last Apex logs
08:57:43 <JuhaHaapa> jose_lausuch: should we pass our Config-Guide notes to you (once finished) or directly through gerrit?
08:57:57 <morgan_orange> we are already trying to trigger security_scan from jenkins
08:58:03 <jose_lausuch> lhinds: you got my +2 already
08:58:14 <lhinds> thx jose_lausuch I saw that
08:58:16 <morgan_orange> the thing is that today in exec we have just a comment line
08:58:23 <jose_lausuch> JuhaHaapa: I think gerrit is fine, you can always amend the changes :)
08:58:25 <morgan_orange> commented line
08:58:26 <lhinds> I am on irc all day, so hit me up if anything fails
08:58:34 <lhinds> I can even jump on a call if need be
08:58:40 <morgan_orange> we also try so do not hesitate to ping us
08:58:41 <jose_lausuch> lhinds: have you tried that code already?
08:58:45 <jose_lausuch> because there were some bugs
08:59:06 <lhinds> jose_lausuch: I have been running it from the jumpbox, same place as where the container runs from.
08:59:16 <jose_lausuch> some of them were detectable thorugh flake8, like oscap and the self.home
08:59:38 <lhinds> yep, well spotted, and thanks for refact'ing that for me
08:59:42 <jose_lausuch> lhinds: can you also update the line in the exec script?
08:59:45 <jose_lausuch> if needed?
08:59:48 <lhinds> that is a lesser used case, so I never saw it
08:59:49 <jose_lausuch> morgan_orange: is that what you mean?
08:59:54 <morgan_orange> yep
09:00:07 <lhinds> jose_lausuch: of course
09:00:30 <morgan_orange> I would suggest
09:00:45 <morgan_orange> 1) on Intel POD7, from the Functest Master docker file, run manually secrutiy_scan
09:00:47 <morgan_orange> when ok
09:00:50 <morgan_orange> 2) complete https://git.opnfv.org/cgit/functest/tree/ci/exec_test.sh
09:01:11 <morgan_orange> Line 137
09:01:24 <jose_lausuch> +1
09:01:28 <lhinds> morgan_orange: sounds good to me.
09:01:30 <morgan_orange> if you have problem to connect to Functest docker file, let us know
09:01:39 <morgan_orange> ok we are already a bit overtime
09:01:45 <morgan_orange> any other topic you would like to share?
09:01:48 <SerenaFeng> sorry
09:01:49 <SerenaFeng> we want to add our pod zte-pod1 in kibana dashboard, I send email to juraj last week, but for now no response yet
09:02:10 <SerenaFeng> Someone else can touch him for me?
09:02:23 <morgan_orange> #info question on Kibana dashboard update (no feedback from juraj)
09:02:43 <SerenaFeng> and I don't have Kibana dashboard access
09:02:55 <morgan_orange> SerenaFeng: I think we may have a problem we already faced...Juraj is probably very busy with vpp integration
09:03:08 <morgan_orange> he planned a session during the summit but may have not time to do it
09:03:15 <morgan_orange> the code has been put on releng
09:03:23 <SerenaFeng> where?
09:03:41 <SerenaFeng> maybe we can try to fix it by ourself
09:03:45 <morgan_orange> #link https://git.opnfv.org/cgit/releng/tree/utils/test/scripts
09:03:56 <morgan_orange> create_kibana_dashboards.py
09:04:00 <SerenaFeng> and where can I get access to Kibana dashboard?
09:04:22 <morgan_orange> you mean the one on testresults.opnfv.org?
09:04:37 <SerenaFeng> yes
09:04:54 <SerenaFeng> no sign up button
09:04:57 <morgan_orange> Maybe I can create an account for you on this machine
09:05:13 <morgan_orange> so you will have access to the test API and to Kibana backend
09:05:33 <SerenaFeng> ok, thank you
09:05:33 <morgan_orange> #action morgan_orange grant access to testresults.opnfv.org to SerenaFeng
09:05:43 <morgan_orange> ok
09:05:53 <morgan_orange> so let's finish now, have a good week
09:05:56 <morgan_orange> #endmeeting