14:30:19 <gzhao> #startmeeting lithium irc weekly sync
14:30:19 <odl_meetbot> Meeting started Wed Apr 29 14:30:19 2015 UTC.  The chair is gzhao. Information about MeetBot at http://ci.openstack.org/meetbot.html.
14:30:19 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:30:19 <odl_meetbot> The meeting name has been set to 'lithium_irc_weekly_sync'
14:30:43 <gzhao> #topic roll call
14:30:46 <phrobb1> Hey gzhao #chair me and I'll help
14:31:11 <gzhao> #chair phrobb1
14:31:11 <odl_meetbot> Current chairs: gzhao phrobb1
14:31:11 <phrobb1> #info Project Leaders and TSC members please #info in
14:31:52 <phrobb1> quiet group….
14:31:54 <Prem> #info Prem for VPNService
14:32:03 <regXboi> #info regXboi for neutron northbound
14:32:09 <LuisGomez> #info LuisGomez for integration
14:32:13 <Shreyas> #info shreyas for Persistence Store Service
14:32:14 <lori> #info lori for lispflowmapping
14:32:24 <gzhao> #info gzhao release
14:33:49 <ttkacik1> #info ttkacik for yangtools and controller
14:34:05 <ttkacik1> #undo
14:34:16 <ttkacik1> #info ttkacik1 for yangtools and controller
14:36:15 <odl-casey> #info odl-casey for OpenDaylight
14:36:26 <lori> ttkacik1: I think #undo only works for chairs
14:36:28 <gzhao> #topic agenda
14:36:47 <gzhao> anyone has any topic suggestion?
14:37:02 <Prem> M4 and M5 related queries
14:37:22 <tbachman> #info tbachman for GBP
14:37:54 <lori> gzhao: I have an issue with empty karaf that I would like to see if others experience too
14:38:01 <phrobb1> #info gzhao I suggest we cover the Weather Page and also get a status from you on any projects that are lagging/behind/non-reporting in the M4 status
14:39:34 <phrobb1> lori:  Please go ahead and start with your topic.  Can you explain a bit what issue you are having with empty karaf
14:39:50 <phrobb1> #topic Empty Karaf issues
14:39:52 <gzhao> #info document status and Karaf 3.0.3 testing are two topics
14:40:30 <gzhao> #topic Weather
14:40:53 <lori> when we build the LISPflowmapping local karaf distribution, we autoload our user facing feature
14:41:19 <lori> this used to work well
14:41:45 <lori> now I get Authentication module unavailable error when accessing RESTCONF
14:41:48 <lori> however
14:41:54 <lori> if I disable autoloading
14:41:59 <lori> start karaf
14:42:02 <lori> wait a bit
14:42:15 <lori> and then load the same feature manually, I don't get that problem
14:42:24 <lori> loaded features are the same in both cases
14:42:28 <lori> same bundles too
14:42:35 <lori> so I guess it's an ordering issue
14:42:58 <lori> karaf with no feature loaded throws an excpetion
14:43:05 <gzhao> lori is that since Karaf 3.0.3 or you think may cause by other issue
14:43:08 <LuisGomez> lori, i think i have seen similar issue in CI system test
14:43:11 <lori> Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException: Could not start JMX connector server
14:43:42 <lori> LuisGomez: I'm not 100% sure, since this is with new functionality
14:43:57 <lori> not well tested before 3.0.3
14:44:18 <LuisGomez> restconf is unavailable for some projects system test and i think it could be same reason
14:44:24 <gzhao> #info lori and LuisGomez saw some ordering issue when load karaf features
14:44:41 <lori> so I couldn't tell if it has anythiong to do with the upgrade
14:45:40 <lori> but it happens on Mac and Linux systems both
14:45:51 <lori> I guess no one was hit by this then
14:46:24 <phrobb1> …except for Luis and the Integration team
14:46:40 <LuisGomez> lori, according to the system test record your feature loadind used to work fine till April 24th
14:46:42 <LuisGomez> https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/lispflowmapping-csit-1node-cds-all-only-master/
14:47:02 <LuisGomez> after that i see failure you mention
14:47:07 <phrobb1> LuisGomez:  are you planning to triage the issue further soon?
14:47:33 <LuisGomez> we need to do something for sure yes
14:48:22 <LuisGomez> i know the symptoms which is restconf not available, i think it could be something to do with aaa
14:48:46 <gzhao> do we have anyone from AAA?
14:48:48 <LuisGomez> i will need to send a broad mail to see if anybody has any idea
14:49:23 <phrobb1> LuisGomez: that makes sense… Is this something we should put on the Weather page since it is affecting multiple projects?
14:49:30 <LuisGomez> yes
14:49:38 <LuisGomez> definetely
14:49:39 <ttkacik1> LuisGomez & lori: does restconf not available happen only for lispflowmapping or other projects too?
14:50:04 <LuisGomez> ttkacik1, other projects too
14:50:17 <lori> ttkacik1: I haven't checked other projects, and in case of lisp the workaround is to load the feature manually
14:50:28 <gzhao> #action LuisGomez and lori  will send board email about the restconf not accessible issue and put on weather page.
14:50:35 <LuisGomez> https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/usc-csit-1node-cds-channel-only-master/
14:50:45 <LuisGomez> usc project seems to run similar issue
14:50:57 <regXboi> LuisGomez: should that be #linked?
14:50:58 <alagalah> #info alagalah GBP
14:51:04 <LuisGomez> but for longer time
14:51:06 * alagalah apologies for being late
14:51:13 <phrobb1> Lori and LuisGomez, have either of you opened a bug on the issue yet?
14:51:17 <phrobb1> welcome alagalah
14:51:22 <edwarnicke> lori: Scanning your description of the karaf issue
14:51:22 <lori> phrobb1: not yet
14:51:33 <phrobb1> welcome edwarnicke
14:51:42 <edwarnicke> lori: Question... are you running more than one copy of ODL locally?
14:51:50 <lori> edwarnicke: no
14:51:59 <lori> edwarnicke: csit test fails too
14:52:09 <lori> edwarnicke: https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/lispflowmapping-csit-1node-cds-all-only-master/225/consoleFull
14:52:12 <edwarnicke> lori: Building
14:52:26 <lori> edwarnicke: this seems relevant: org.eclipse.jetty.security.ConstraintSecurityHandler@7967ae7a: java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@1e8626f
14:52:36 <LuisGomez> both lisp and usc have odl-restconf in the list of features to install, however it does not work
14:53:09 <edwarnicke> LuisGomez: Hmm... that's weird... is it *only* those two projects, and not others that use restconf?
14:53:26 <LuisGomez> and there is 3rd case, vtn when it loads all features too:
14:53:29 <LuisGomez> https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/vtn-csit-1node-cds-manager-all-master/
14:53:43 <gzhao> #info LuisGomez says both lisp and usc have odl-restconf in the list of features to install, however it does not work
14:53:48 <edwarnicke> Building lispflowmapping to see what I can see
14:54:06 <edwarnicke> lori: LuisGomez What is the error your get from restconf ?
14:54:30 <LuisGomez> 503 Server Unavailable
14:54:40 <lori> Authentication module not available
14:54:41 <LuisGomez> there is exception in karaf too
14:54:44 <LuisGomez> 2015-04-29 13:23:59,066 | ERROR | tp2008964540-792 | ContainerResponse                | 214 - com.sun.jersey.jersey-server - 1.17.0 | Mapped exception to response: 503 (Service Unavailable)
14:54:52 <edwarnicke> LuisGomez: OK
14:55:02 <edwarnicke> LuisGomez: I am suspicious about aaa... but lets see
14:55:26 <gzhao> due to time constrains, anyone has any other topic?
14:55:33 <LuisGomez> edwarnicke, i was thinking about the same
14:55:51 <gzhao> #topic documentation
14:56:40 <gzhao> #link https://lists.opendaylight.org/pipermail/release/2015-April/002080.html <-- PTL in mentioned projects, please send gerrit URL for docs.
14:56:46 <ttkacik1> (16:54:43) LuisGomez: 2015-04-29 13:23:59,066 | ERROR | tp2008964540-792 | ContainerResponse                | 214 - com.sun.jersey.jersey-server - 1.17.0 | Mapped exception to response: 503 (Service Unavailable)
14:56:57 <ttkacik1> that effectivelly means md-sal was not up yet
14:57:11 <ttkacik1> if this was from restconf
14:57:52 <LuisGomez> ttkacik1, this error show for 10 mins after you start karaf with the features pre-installed
14:58:09 <ttkacik1> going to try to reproduce
14:58:09 <LuisGomez> after 10 mins i fail the test for timeout reasons
14:58:59 <ttkacik1> LuizGomez: last time lispflowmapping worked was 24th?
14:59:03 <LuisGomez> https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/lispflowmapping-csit-1node-cds-all-only-master/225/consoleFull
14:59:48 <LuisGomez> yes, you can see here: https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/lispflowmapping-csit-1node-cds-all-only-master/
15:00:55 <LuisGomez> usc and vtn were failing for longer though
15:01:22 <edwarnicke> This issue appears to be this:
15:01:25 <edwarnicke> https://www.irccloud.com/pastebin/CA4QB7zS
15:02:41 <gzhao> LuisGomez: we will investigate USC
15:03:51 <lori> edwarnicke: I'll try removing our neutron feature to see if it helps
15:04:27 <gzhao> just to interrupt a little, does anyone has any other issue? we are over our time.
15:05:39 <gzhao> phrobb1: LuisGomez: edwarnicke ttkacik1 do you want this meeting to continue to record, or we can do that in separate email
15:05:52 <LuisGomez> i will open a bug in integration and will send mail for further discussion
15:06:36 <phrobb1> Yes agree with LuisGomez, let's close this meeting if no other topics and continue karaf/neutron issue in a separate thread
15:06:37 <gzhao> #info edwarnicke points the error might be https://www.irccloud.com/pastebin/CA4QB7zS
15:06:53 <gzhao> 3,
15:06:56 <gzhao> 2
15:07:03 <gzhao> 1
15:07:09 <gzhao> #endmeeting