08:01:03 <morgan_orange> #startmeeting Functest weekly meeting July 12th
08:01:03 <collabot> Meeting started Tue Jul 12 08:01:03 2016 UTC.  The chair is morgan_orange. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:01:03 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
08:01:03 <collabot> The meeting name has been set to 'functest_weekly_meeting_july_12th'
08:01:08 <morgan_orange> #topic call role
08:01:54 <morgan_orange> #info Morgan Richomme
08:01:54 <ollivier> #info ollivier
08:02:04 <morgan_orange> #chair jose_lausuch ollivier
08:02:04 <collabot> Current chairs: jose_lausuch morgan_orange ollivier
08:02:42 <morgan_orange> #info agenda: https://wiki.opnfv.org/display/functest/Functest+Meeting
08:02:49 <SerenaFeng> #info SerenaFeng
08:02:58 <jose_lausuch> #info Jose Lausuch
08:03:01 <morgan_orange> #topic action point follow up
08:03:07 <jose_lausuch> nice to use this chan :)
08:03:11 <morgan_orange> #link http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2016/opnfv-testperf.2016-07-05-08.00.html
08:03:36 <morgan_orange> #info AP1 => request sent to helpdesk to get our gerrit robot on this chan, no feedback so far
08:03:55 <morgan_orange> #info but jenkins/flake8 now operationnal for Functest
08:04:32 <morgan_orange> #info AP2: document opnfv-functest documented from the functest main page, I should also still change it in the different documents
08:04:57 <morgan_orange> #info AP3: ollivier pythonization of ODL almost done
08:05:22 <morgan_orange> #no other action points
08:05:24 <jose_lausuch> shall we fix the .sh problem or we wait for pythonization?
08:06:00 <ollivier> I think we should fix .sh anyway.
08:06:07 <jose_lausuch> ya
08:06:15 <morgan_orange> we should avoid CI blocking, so I would +1 for the fix and/or disable ODL testing in CI
08:06:21 <jose_lausuch> trozet reported the bug to jira
08:06:32 <morgan_orange> yep we will discuss all that in next sections
08:06:44 <ollivier> Are you speaking about the admin rights of *.sh?
08:06:49 <morgan_orange> #topic Colorado release Status
08:06:50 <jose_lausuch> yes
08:06:58 <jose_lausuch> or whatever the issue is
08:07:20 <morgan_orange> #info feedback provided to all the installers (except apex / mail ready but not sent yesterday)
08:07:36 <jose_lausuch> /home/opnfv/repos/functest//testcases/Controllers/ODL/start_tests.sh: Permission denied
08:07:39 <morgan_orange> in // tim created JIRA related to the differnt issue on CI pipeline detected with apex
08:07:56 <morgan_orange> #link https://wiki.opnfv.org/display/apex/Apex+Scenario+Status
08:08:03 <morgan_orange> so let's start with apex
08:08:28 <morgan_orange> #info apex scenario: issues with ODL, security_scan and DVR support (odl_l3 scenario)
08:08:29 <ollivier> jose_lausuch: I am checking it in my container.
08:09:24 <jose_lausuch> ok
08:09:35 <jose_lausuch> in mine, it has X permissions
08:09:40 <ollivier> jose_lausuch: It seems fine here (I don't remember modifying it by hand)
08:10:34 <morgan_orange> ollivier: we have access to the LF POD1 (Apex) so if needed we need to find a windows slot but we shall be able to test it directly on the target POD
08:11:41 <jose_lausuch> my connection to LFPOD1 is not stable, I get kicked out in 5 sec
08:11:53 <morgan_orange> we need also to see with luke, security scan are now working but issues with data collection
08:12:06 <morgan_orange> jose_lausuch: ok....
08:12:26 <morgan_orange> regarding the other installers
08:12:51 <morgan_orange> #info fuel feedback most of the scenarios almost OK http://testresults.opnfv.org/reporting/functest/release/master/index-status-fuel.html
08:13:26 <morgan_orange> #info fuel last issues: bgpvpn (integration in progress)
08:13:42 <morgan_orange> we may note that the onos-sfc is already deployed
08:13:54 <morgan_orange> not sure onos-sfc suite is already triggered
08:13:57 <jose_lausuch> is the test working?
08:13:58 <jose_lausuch> mmm
08:14:07 <ollivier> morgan_orange: Ok we have to check it quickly :)
08:14:27 <jose_lausuch> morgan_orange: where do you see that scenario triggered?
08:15:14 <morgan_orange> http://testresults.opnfv.org/test/api/v1/results?case=tempest_smoke_serial&last=10&scenario=os-onos-sfc-ha
08:15:28 <morgan_orange> tested on lf-pod2 10/7
08:15:30 <jose_lausuch> ok here https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/154/console
08:15:53 <jose_lausuch> not triggered, because vping fails
08:16:07 <morgan_orange> #action morgan_orange asks Anthony if onos-sfc tests can be enable as there is one fuel scenario ready to support it
08:16:52 <morgan_orange> https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-master/148/console
08:17:19 <morgan_orange> seems that the tests were already included in onos suite
08:17:25 <morgan_orange> 2016-07-10 11:33:10,878 - sfc_fun_log - INFO - 	The packet has reached VM2 Instance
08:17:26 <morgan_orange> 2016-07-10 11:33:10,879 - sfc - INFO - 		SFC function Working
08:17:38 <morgan_orange> I should discuss how results are collected
08:18:04 <morgan_orange> it seems that the sfc suite remplaced the old one
08:18:27 <morgan_orange> http://testresults.opnfv.org/test/api/v1/results?case=onos&last=10&scenario=os-onos-sfc-ha
08:19:27 <morgan_orange> #info feedback on compass, except ocl scenario (still under integration), other scenarios have issues with rally sanity - but last runs seems OK so wait and see ..
08:19:59 <jose_lausuch> something to fix
08:20:34 <morgan_orange> #info feedback on joid: Narinder indicated a DHCP issue under investigation that should be a root cause of several issues
08:20:50 <lhinds> #info Luke Hinds
08:21:16 <morgan_orange> question: we have our automatic reporting, Tim created a wiki page to reference the JIRA associated with the scenario, I think we should reference the Jira for the other installer here
08:21:46 <morgan_orange> in the wiki page I did not see the Jira created by Viktor and Juha (results on tempest and Rally not reaching the success criteria)
08:22:13 <morgan_orange> #action morgan_orange create wiki pages to reference installer related JIRA
08:22:29 <jose_lausuch> you mean this?   https://wiki.opnfv.org/display/apex/Apex+Scenario+Status   ?
08:22:34 <morgan_orange> yes
08:23:16 <morgan_orange> shall we create a page or ask each installer to do it, assuming that on our side we have our automatic reporting...
08:23:47 <jose_lausuch> mmmm
08:23:59 <jose_lausuch> our automatic reporting tells how functest behaves
08:24:09 <jose_lausuch> but we dont have to check every isntaller' deployment status
08:24:17 <jose_lausuch> I think its something for each installer to do
08:24:20 <jose_lausuch> but its a nice wiki
08:24:26 <jose_lausuch> maybe david mcbride could propose that
08:24:31 <jose_lausuch> but I dont think its for us to do
08:24:39 <morgan_orange> ok agree
08:24:49 <morgan_orange> just reference the links in our Colorado page
08:25:11 <morgan_orange> other topic: I also noticed a bug (I need to create a Jira) on rally_sanity => status is always PASS even the criteria is not correct
08:25:43 <jose_lausuch> ok
08:25:48 <jose_lausuch> can you assign it to juhak?
08:26:01 <morgan_orange> SerenaFeng: as Juha is in vacations may I assign this future Jira to you => need to change values in the DB and fix it in the code (I assume it is because the success rate is given as a string so I do not knwo what the test is doingà
08:26:01 <jose_lausuch> or is he on vacation?
08:26:17 <SerenaFeng> ok
08:26:47 <SerenaFeng> but I will not do it until next week, it that ok?
08:26:56 <morgan_orange> #Action morgan cretae a jira assigned to SerenaFeng and details issues on rally_sanity result
08:27:24 <SerenaFeng> I will attend OpenStackChina Summit from tommorow to the end of this week
08:27:24 <morgan_orange> SerenaFeng: yep no problem, we will keep on being over optimistic until next week, then with the change on the DB we will come back to reality
08:27:45 <SerenaFeng> ok
08:27:45 <morgan_orange> no rush it is just reporting and we have everything in DB
08:28:08 <morgan_orange> other subtopic for Colorado, valentin got lots of question on the famous vIMS test case
08:28:25 <morgan_orange> boucherv_orange: a little status on the different requests so far
08:29:25 <morgan_orange> boucherv_orange: shall we create Jira to track the requests?
08:30:19 <morgan_orange> #topic security_scan
08:30:37 <morgan_orange> #info discussion planned during the TSC meeting today to validate the creation of a dedicated repo
08:31:35 <morgan_orange> 2 things to discuss: 1) JIRA (problem with data collection => as scans are now OK) 2) success criteria
08:31:36 <lhinds> I did give this some time to think over, do you still want to try to define a binary pass / fail grade?
08:31:39 <morgan_orange> regarding 1)
08:32:24 <morgan_orange> I have no strong opinion, as you said it should be more seen as recommendation than test for scenario validation
08:32:48 <morgan_orange> so a binary PASS/FAIL would make sense later when we got more maturity and the system in place for all the installers
08:33:03 <morgan_orange> we can put it in place but not consider it for scenario validation
08:33:25 <morgan_orange> it is however interesting to report it so installer may make some evolution to get better results
08:33:34 <lhinds> What I was thinking is that we could pull out the overall score into functest's dashboard?
08:34:02 <morgan_orange> so my suggestion would be a reporting like Tempest (duration, nb of failed, details on failed cases) but just for information => not used for validation of the scenario
08:34:04 <lhinds> e.g. 48% success, 50% fail, 2% N/A
08:34:23 <lhinds> and then link to the reports (the full html with eye candy)
08:35:13 <morgan_orange> yes we could push the results to the DB as a Tempest result => http://testresults.opnfv.org/test/api/v1/results?case=tempest_smoke_serial&last=2&installer=apex
08:35:22 <morgan_orange> with also the link to the artifact report
08:35:33 <morgan_orange> and based on that re use a jinjer2 framework like vIMS or Tempest
08:35:46 <morgan_orange> http://testresults.opnfv.org/reporting/functest/release/master/index-tempest-apex.html
08:36:07 <morgan_orange> if you are OK with that luke, you can add the push to DB, I can do all the reporting/jinjer2 stuff
08:36:22 <morgan_orange> and add a security tab in the reporting page
08:36:35 <morgan_orange> http://testresults.opnfv.org/reporting/ in addition of tempest vIMS and Status
08:37:31 <lhinds> I don't mind that, looks like there are already a few testcases doing a DB push that I could lift the related functions from, and use in my code
08:37:49 <morgan_orange> it is also not a top priority (as not part of scenario validation)..we must focus on CI integration
08:37:59 <morgan_orange> ok I will create some minor JIRA then
08:38:12 <lhinds> ok, well that would suit me well, as I will be hands full with extending to different installers
08:38:47 <lhinds> sounds good though
08:38:55 <morgan_orange> regarding the JIRA, I hope I will find some time to also test (hope my connection will be more stable...)
08:39:50 <morgan_orange> #topic AoB
08:40:01 <lhinds> last of all on sec_scan
08:40:09 <morgan_orange> SerenaFeng: do you think you could have time to meet people from rally at OpenStack Summit China?
08:40:15 <lhinds> going to start looking into the report download failure today
08:40:24 <morgan_orange> #undo
08:40:24 <collabot> Removing item from minutes: <MeetBot.ircmeeting.items.Topic object at 0x2cd87d0>
08:40:37 <lhinds> will chat with Tim later on (although he might be on leave)
08:40:43 <morgan_orange> I try to catch some stuff for the minutes
08:40:49 <SerenaFeng> If need, I will try to contact them
08:40:54 <morgan_orange> #info security_scan now producing nice reports
08:41:07 <lhinds> so you can hashtag an action against me that I am on it
08:41:12 <morgan_orange> #info still an issue with integration in CI pipeline when collecting the restuks
08:41:33 <morgan_orange> #info secruity_scan not used for scenario validation (must be generalized to all the installers first)
08:42:03 <morgan_orange> #info however as teh info are usefull => adopt tempest approach to collect success rate, errors and create a Security reporting (like Tempest of vIms)
08:42:29 <morgan_orange> #info minor priority / generalization to othe installers and fix in CI
08:42:39 <morgan_orange> #topic AoB
08:43:17 <SerenaFeng> Some of my collegues works in rally will be there, maybe I can contact them
08:43:30 <morgan_orange> #info SerenaFeng participating to OpenStack Summit China => could be interesting to discuss with Rally community as we are using rally for Tempest/Rally and could use it more for additional scenario
08:44:18 <morgan_orange> ollivier: started working really more upstream with ODL (we are wating for their merge on the last patch)
08:44:34 <SerenaFeng> about what information you want me to disscuss with them?
08:44:50 <morgan_orange> I think we should investigate to do the same with rally (I know that Juha and Viktor already reported bugs but it could be interesting to discuss also the feature evolutions)
08:45:50 <morgan_orange> the idea is to get a better understanding how it works, where can we discuss feature evolution, what they could expect from us as end to end tester in OPNFV ecosystem
08:46:16 <morgan_orange> what kind of contribution we could reasonably make
08:46:18 <morgan_orange> ...
08:46:24 <morgan_orange> another topic for AoB
08:46:39 <morgan_orange> we start getting questions from feature projects aiming to connect the compute/control node
08:47:01 <SerenaFeng> okey, I will have a try to disscuss with them
08:47:01 <morgan_orange> basically for feature projects tehre are several ways to be onboarded of teh SUT
08:47:12 <morgan_orange> the plugin/charm/addon/ansible playbook in the installer
08:47:22 <morgan_orange> postinstallation scripts
08:47:53 <morgan_orange> some projects seem to priviledge the second approach but today we do not have an abstraction layer for such connectivity and have to do it manually installer per installer
08:48:10 <jose_lausuch> let's propose it for D-release
08:48:18 <jose_lausuch> it shouldnt be that difficult
08:48:21 <morgan_orange> for D release, we probably have to think to that, see if there is already something for apex in SNAPS framework
08:48:30 <jose_lausuch> 4 if-else statements :)
08:48:33 <morgan_orange> it should not be very difficult
08:48:48 <morgan_orange> the question is is it good to encourage this approach versus a more installer integrated approach
08:49:08 <morgan_orange> it is more a question for genesis
08:49:16 <jose_lausuch> the ideal case would be to have an installer API
08:49:20 <jose_lausuch> common for all installers
08:49:24 <jose_lausuch> but yes, thats genesis
08:49:37 <jose_lausuch> but yet another "requirement"
08:49:47 <jose_lausuch> which might stay in JIRA forever
08:50:23 <morgan_orange> OK maybe we can also discuss that with the test community because whatever the nature of the tests, such API/library would be useful
08:50:51 <jose_lausuch> yes
08:50:56 <morgan_orange> #info requests from feature project to be able to connect to the different SUT nodes
08:51:12 <morgan_orange> #info so far no standard way => namual processing different from one installer to another
08:51:34 <morgan_orange> #info discussions on an installer integrated approach versus a post-installation through test umbrella projects
08:52:03 <morgan_orange> #info could be a genesis topic
08:52:05 <morgan_orange> #action morgan_orange jose_lausuch discuss this topic with testing community
08:52:11 <jose_lausuch> AoB: enjoy this video https://www.youtube.com/watch?v=Zzb0xLfnn9Y
08:52:14 <jose_lausuch> ok
08:52:50 <morgan_orange> #info summer organization: I will be on vacations on Wednesday until the 8th of August
08:53:15 <morgan_orange> #info Jose will take the lead until beginning of August
08:53:27 <morgan_orange> we need someone to lead for the first week of August
08:53:31 <jose_lausuch> starting tomorrow?
08:53:39 <morgan_orange> yes....
08:53:42 <jose_lausuch> ok
08:54:08 <jose_lausuch> who volunteers for august?
08:54:19 <SerenaFeng> envy, no summer vacation in China
08:54:28 <morgan_orange> SerenaFeng: would you be OK to represent Functest? for the first week of August
08:55:09 <SerenaFeng> the weekly meeting?
08:55:34 <morgan_orange> yes weekly meeting
08:55:39 <morgan_orange> release meeting (3Pm UTC)  are not convenient for China but you may report to David Mc Bride by mail (idem for the weekly test meeting -2 PM UTC)
08:56:38 <SerenaFeng> okey
08:56:44 <morgan_orange> #info SerenaFeng will lead Functest on the first week of August
08:56:55 <morgan_orange> any other info you want to share this week?
08:57:08 <jose_lausuch> not from my side
08:57:23 <SerenaFeng> I have implement auto update of testapi using ansible-playbook
08:57:34 <SerenaFeng> and make the git review
08:57:48 <morgan_orange> #info auto update of testapi using ansible-playbook impemented by SerenaFeng
08:58:02 <morgan_orange> is it enable on testresults.opnfv.org?
08:58:16 <SerenaFeng> until now, I think test-api evoluation is completely finish for now
08:58:32 <SerenaFeng> yes
08:59:00 <morgan_orange> the PUT pethod is tehre, I did not use it yet - we probably need to code some function in functestUtils getTrustIndicator, modifyTrustIndicator
08:59:10 <SerenaFeng> you can run the script in local vm and update testapi of testresults.opnfv.ofg
08:59:10 <morgan_orange> but great work on the test API
08:59:46 <SerenaFeng> do we need to modify the date in the database?
08:59:53 <raghavendrachari> hi morgan , amwaitign for the doc ?
09:00:36 <morgan_orange> SerenaFeng:  which date?
09:00:38 <SerenaFeng> The update_mongodb.py already support the modification, only need to add the changes to changes_in_mongodb.py
09:00:47 <SerenaFeng> trust_indicator
09:00:56 <SerenaFeng> to support the new data model
09:01:23 <morgan_orange> yes you want to do or I do, as you want
09:01:38 <SerenaFeng> from trust_indicator: 0 to trust_indicator:{current: 0, history: []}
09:02:13 <SerenaFeng> I add a jira to implement it
09:02:22 <morgan_orange> raghavendrachari: the patch on devguide has been submitted the doc is here http://artifacts.opnfv.org/functest/docs/devguide/index.html
09:02:33 <raghavendrachari> ok
09:03:36 <morgan_orange> #action SerenaFeng update test api on testresults.opnfv.org with last version of datamodel
09:04:58 <morgan_orange> ok that is all for this week
09:05:27 <morgan_orange> have a good week, I will be back on the 8th of August, enjoy the Alicante style
09:05:48 <morgan_orange> #endmeeting