16:00:02 <frankbrockners> #startmeeting OPNFV BGS daily Arno release readiness synch
16:00:02 <collabot> Meeting started Fri May 22 16:00:02 2015 UTC.  The chair is frankbrockners. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:02 <collabot> The meeting name has been set to 'opnfv_bgs_daily_arno_release_readiness_synch'
16:00:12 <frankbrockners> #info Frank Brockners
16:01:03 <morgan_orange> #info Morgan Richomme
16:01:14 <frankbrockners> Hey Morgan
16:01:52 <trozet> #info Tim Rozet
16:02:11 <frankbrockners> will probably be a quick meeting today - Jonas and Peter can't make it...
16:02:32 <frankbrockners> so we can get started - and record status for others.
16:02:44 <frankbrockners> I have a brief update from Jonas... will info this in
16:02:54 <frankbrockners> #info Updates on POD1 autodeploy
16:03:17 <frankbrockners> #info An non HA deployment is in place at LF POD1.
16:03:53 <frankbrockners> #info Jonas and team are finishing up some stuff for the ODL tests, hope to have it on Sunday. (per Jonas)
16:04:18 <jose_lausuch> #info Jose Lausuch
16:04:20 <frankbrockners> that's it for Fuel - so POD1 is switched from pure O/S to O/S with ODL now
16:04:58 <frankbrockners> #info Updates on POD2
16:05:12 <frankbrockners> trozet - do you want to give a brief update?
16:05:36 <trozet> #info I updated the quotas this morning, and tested it manually.  Kicked off a new daily master so we should see the results with the quotas bumped
16:06:04 <trozet> #info looking into functest failures with morgan_orange and jose_lausuch
16:06:18 <trozet> thats it
16:06:30 <frankbrockners> did "network_external" help with any of the tests?
16:06:46 <frankbrockners> i.e. external network being available now?
16:06:57 <trozet> frankbrockners: you have to tell tempest what the external network is
16:06:59 <trozet> and tell it to use it
16:07:15 <trozet> I'm trying to determine if rally does that automatically or if we need to update the tempest.conf file
16:07:17 <frankbrockners> ok... sounds like we should move to Morgan and Jose :-)
16:07:25 <frankbrockners> #info Updates on Functest
16:07:39 <frankbrockners> morgan_orange or jose_lausuch?
16:07:50 <morgan_orange> #info last night tests shared; results stable on POD1 (same errors reproducible)
16:07:52 <jose_lausuch> the thing is that for fuel deployment, the external network is created and taken by rally to put it in the tempest.conf, at least this is what I can tell
16:08:17 <morgan_orange> #info more errors on POD2 but more tests attempted..
16:08:28 <morgan_orange> #vPing failed on POD2 'Ping not seen)
16:08:37 <morgan_orange> #info vPing failed on POD2 'Ping not seen)
16:08:42 <morgan_orange> #info vPing OK on POD1
16:09:33 <frankbrockners> morgan_orange: was this done on the new O/S + ODL setup or the old O/S only setup (with native neutron networking)
16:09:34 <morgan_orange> #info ODL tests KO on POD1 (problem of rights on the file)
16:10:37 <jose_lausuch> maybe we need to set the permissions from jenkins or something, strange error
16:11:05 <morgan_orange> it was on https://build.opnfv.org/ci/job/functest-opnfv-jump-1/16/console
16:11:10 <morgan_orange> #link https://build.opnfv.org/ci/job/functest-opnfv-jump-1/16/console
16:11:20 * fdegir jose_lausuch I'll look into those permission issues and come back to you
16:11:21 <morgan_orange> #info ODL test not executed on POD2 (vPing failure)
16:11:31 <jose_lausuch> sure, thanks
16:11:34 <morgan_orange> I push a patch mentioned by Peter
16:11:43 <jose_lausuch> ah
16:11:45 <morgan_orange> call bash start_tests.sh
16:11:54 <jose_lausuch> but will that help?
16:11:56 <morgan_orange> to be merged on releng
16:12:30 <fdegir> will merge it whenever the patch comes in
16:13:13 <frankbrockners> from what I know Peter mentioned that manual ODL tests on POD2 ran ok
16:13:16 <morgan_orange> it has been done by aric
16:13:28 <fdegir> ok
16:13:41 <morgan_orange> yes ODL manual run on POD2 was OK
16:13:53 <morgan_orange> they have not been executed last night due to vPing Timeout
16:14:02 <morgan_orange> it seemsn that both VM properly started
16:14:20 <morgan_orange> but the no ping has been captured in the nova console
16:14:41 <jose_lausuch> but I pushed a patch to timeout vping
16:14:45 <jose_lausuch> did ntit work?
16:14:51 <morgan_orange> yes it worked well
16:14:56 <jose_lausuch> ah ok
16:14:58 <morgan_orange> the test did not loop
16:15:04 <jose_lausuch> great
16:15:04 <morgan_orange> it exited after time out
16:15:15 <morgan_orange> https://build.opnfv.org/ci/view/functest/job/functest-opnfv-jump-1/16/console
16:15:28 <jose_lausuch> ah, ok, so then it does not continue with the next tests
16:15:52 <frankbrockners> what are the remaining tasks from a functest perspective? trozet changed the quotas - which hopefully fixes some issues. what is the plan to get rally recognize the external network on POD2 correctly (to drive the tempest.conf)?
16:16:22 <morgan_orange> but raceback (most recent call last):
16:16:22 <morgan_orange> File "/home/jenkins-ci/.functest/vPing/vPing.py", line 256, in <module>
16:16:22 <morgan_orange> main()
16:16:22 <morgan_orange> File "/home/jenkins-ci/.functest/vPing/vPing.py", line 247, in main
16:16:22 <morgan_orange> if EXIT_CODE == 0:
16:16:22 <morgan_orange> UnboundLocalError: local variable 'EXIT_CODE' referenced before assignment
16:16:23 <morgan_orange> Build step 'Execute shell' marked build as failure
16:16:56 <morgan_orange> remaining tasks
16:17:23 <morgan_orange> - have a complete run (even with some errors)
16:17:35 <morgan_orange> - reduce the number of errors (quotas should fix some errors)
16:18:02 <morgan_orange> but last night we got new errors we never had Invalid credentials on tempest.api.identity.admin.v3
16:18:14 <morgan_orange> - document the errors
16:18:51 <morgan_orange> we still have lots of errors on nova/neutron bench but quota could be also one solution
16:19:25 <morgan_orange> we could contact Rally team next week because I think there are cleaining issues that could also be the cause of some errors
16:19:57 <morgan_orange> #link https://wiki.opnfv.org/r1_tempest?&#error_follow_up
16:19:57 <trozet> identity stuff is set in tempest config
16:20:16 <frankbrockners> let's info some of this ... (doing this right now)
16:20:30 <trozet> #info I think we need to save the tempest config file from runs.  It is hard to debug these problems without it
16:20:51 <morgan_orange> yes but it is weird, if it would use identity creds from this config file everything shall fail..we source the o/S creds before launchin the installation
16:21:05 <morgan_orange> this error appear only on both POD yesterday night
16:21:18 <frankbrockners> #info FuncTest remaining tasks (a) achieve a complete test run (even with some errors), (b) reduce the number of errors (we still have lots of errors on nova/neutron bench but quota could be also one solution) (c) document the errors
16:21:55 <morgan_orange> the config file is not cleaned after test, it is always available on the Jumphost
16:22:04 <frankbrockners> back on the earlier question: is there a way to feed rally the info on the external network and other tempest config parameters?
16:22:53 <trozet> frankbrockners: the suggestion is rally figures it out for you
16:22:56 <morgan_orange> my view was that tempest config was dynamically created based on a static default config and dynamic information retrieve through the different client
16:22:57 <trozet> http://pastebin.com/VTw4xyG2 line 62
16:23:06 <trozet> that is the tempest config for pod1
16:23:16 <trozet> I need the tempest config for pod2 to help debug these issues
16:23:20 <jose_lausuch> morgan_orange: but it should, if the test go well, at the end it will execute a clean, which will delete .rally/
16:23:37 <morgan_orange> trozet: we will get it but I cleaned functest for the new run
16:23:51 <trozet> morgan_orange: np we will see it I guess when functest runs on daily master
16:23:58 <morgan_orange> oups you are right, if the test go until the end we will clean it..
16:25:44 <morgan_orange> so frankbrockners I think for the last 2 weeks we will need support on tempest/rally tuning...
16:26:15 <frankbrockners> quickly on tempest.conf: could you push it to some git - or log the tempest.config file to console?
16:26:37 <jose_lausuch> that is actually a good idea
16:26:45 <jose_lausuch> we could output it from the jenkins job
16:28:29 <trozet> +1
16:28:31 <frankbrockners> morgan_orange: ack on the need to have support - though not sure how to get it. Should we ask opnfv-tech-discuss again for help? And all of us can dig for tempest/rally experts in our companies for help...
16:29:19 <trozet> just copy it
16:29:24 <trozet> as part of the jenkins bash script
16:29:49 <jose_lausuch> copy it where?
16:30:59 <morgan_orange> in console
16:31:01 <morgan_orange> cat .rally/tempest/for-deployment-*/tempest.conf
16:31:10 <morgan_orange> or in final result dir?
16:31:15 <jose_lausuch> jap
16:31:37 <jose_lausuch> thats another option, or even both
16:31:55 <trozet> yeah you can just cat it into the log file
16:32:06 <trozet> or copy it into /tmp
16:32:07 <trozet> or whatever
16:32:44 <frankbrockners> morgan_orange: on additional help: have you tried asking iben for help?
16:33:12 <morgan_orange> we could also add a rally show networks in the console (as we do for image , flavours and services)
16:33:27 <morgan_orange> frankbrockners: not doen nominatively
16:33:40 <morgan_orange> I will ask him
16:34:06 <frankbrockners> thanks - he might also know other test folks who could help
16:34:10 <morgan_orange> people from Mirantis on rally-chan were also responsive when I started discovering Rally
16:34:42 <morgan_orange> we will see on Tuesday..
16:35:40 <frankbrockners> some admin-stuff: Monday is memorial day in the US and pentecoast in several countries
16:36:28 <frankbrockners> plan was to still do a synch on Monday (Peter Bandzi hosts it - I'm traveling) - we'll see who is going to make it (might be a very short meeting)
16:36:55 <frankbrockners> trozet: Is dradez covering for you next week?
16:37:13 <trozet> dradez and dfarrell07 is also around to help
16:37:18 <trozet> radez*
16:37:22 <frankbrockners> perfect - thanks
16:37:38 <trozet> I'll still be checking email
16:38:46 <frankbrockners> ok - are we done for today?
16:39:28 <frankbrockners> looks like it..
16:39:31 <jose_lausuch> yep
16:39:43 <frankbrockners> Everyone have a nice loooong weekend
16:39:49 <frankbrockners> #endmeeting