22:01:33 <jamoluhrsen> #startmeeting integration
22:01:33 <odl_meetbot> Meeting started Thu Dec  6 22:01:33 2018 UTC.  The chair is jamoluhrsen. Information about MeetBot at http://ci.openstack.org/meetbot.html.
22:01:33 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:33 <odl_meetbot> The meeting name has been set to 'integration'
22:04:02 <dfarrell07> #info pybot CLI has been deprecated, moved to robot CLI
22:07:38 <jamoluhrsen> #info abelur notes that most infra issues seem to be resolved except the IO issue and that might mostly be related to our log backup job especially since our logs are mostly a bunch of small files
22:11:06 <jamoluhrsen> #info dfarrell07 notes that we may get budget for interns that we can use to help trim the jobs down to help save money
22:13:02 <jamoluhrsen> #info LuisGomez wondering if we have any jobs that build docker images for ODL. we do have the jobs but they were not maintained for a year or so. They probably need some love to get back working
22:25:33 <vorburger> jamoluhrsen: LuisGomez: hey I'm about to be very interested in that (jobs that build docker images for ODL) as well... keep in touch? actually, watch this issue:
22:26:04 <vorburger> jamoluhrsen: LuisGomez: dfarrell07: https://jira.opendaylight.org/browse/COE-30
22:27:07 <dfarrell07> vorburger: I've had logic for years and years to build containers/VMs using Packer, using Ansible-ODL under the hood
22:27:23 <dfarrell07> vorburger: but no one was using them so I haven't really been maintaining recently
22:28:05 <dfarrell07> vorburger: I also tried to make a job to do this on Jenkins and push to Dockerhub, but ran into blocking infra limits
22:28:15 <dfarrell07> vorburger: I have an issue, let me find it
22:28:19 <vorburger> dfarrell07: well looks like that's about to come in handy again, for (apparently) both LuisGomez and me. In fact, can I internally ping you for COE-30 some time? Possibly more of a Jan than Dec issue.
22:28:50 <dfarrell07> vorburger: yeah, perfect, I'd love a good reason to be able to give cycles to that logic
22:28:59 <vorburger> dfarrell07: FYI in last week's TSC, zxiiro said that whenever we (CoE folks) were ready and really wanted this, to open a helpdesk ticket and he/someone would assist from the infra side
22:29:36 <vorburger> dfarrell07: sounds like a plan. It's not something for tomorrow morning as I haven't even got to being able to build that image locally, and I first need to progress on where I actually use that image - but I'll get back to you!
22:31:18 <dfarrell07> vorburger: cool, just let me know when you're ready :)
22:31:21 <jamoluhrsen> #action LuisGomez to email LF folks to reduce our number of parallel jobs to see if we can get more reliable jobs
22:31:23 <dfarrell07> vorburger: I linked this issue to yours https://jira.opendaylight.org/browse/INTPAK-12
22:33:09 <abelur> #info https://gerrit.linuxfoundation.org/infra/#/c/13828/
22:34:44 <jamoluhrsen> LuisGomez: https://git.opendaylight.org/gerrit/#/c/78490/
22:37:10 <jamoluhrsen> #action jamoluhrsen to test zxiiro's draft patches for global-jjb macros in the sandbox
22:52:19 <LuisGomez> vorburger, count on me as well
22:53:02 <LuisGomez> i was not aware of COE-30 but I can definitely spend some cycles there too
00:29:38 <zxiiro> jamoluhrsen: dfarrell07 looks like the bad URLs from autorelease is fixed https://jenkins.opendaylight.org/releng/job/integration-distribution-test-neon/118/parameters/
00:29:54 <zxiiro> let me know if you still see anything broken
00:29:56 <jamoluhrsen> zxiiro: yeah. looks fine now. I was checking it out earlier
00:29:59 <jamoluhrsen> thanks for that
17:05:42 <jamoluhrsen> LuisGomez: did you forget to mark this OKAY?   https://docs.google.com/spreadsheets/d/1dYOY99twqHV_Q0YorAOOxmL0aFc3icNXg8qA_zGwKyA/edit#gid=1286339548
17:56:09 <jamoluhrsen> zxiiro: abelur: LuisGomez: is every job failing right now to upload logs?
17:57:02 <LuisGomez> just updated jamoluhrsen
18:54:59 <jamoluhrsen> LuisGomez: zxiiro: abelur: dfarrell07: CSIT is broken. patch submitted and email sent.
18:59:39 <LuisGomez> ok jamoluhrsen
22:08:26 <abelur> LuisGomez: jamoluhrsen do you know why we dont have these files anymore under Oxygen?
22:09:06 <LuisGomez> which files?
22:09:09 <abelur> integration/distribution/opendaylight/pom.xml?
22:09:42 <LuisGomez> in oxygen there was no "common" distribution
22:09:53 <LuisGomez> that started in fluorine
22:11:20 <abelur> LuisGomez: so we dont have the integration/dist project in AR for oxygen?
22:11:52 <LuisGomez> only the opendaylight/pom.xml is missing
22:12:09 <LuisGomez> the rest should be there
22:13:46 <LuisGomez> just to be clear, the opendaylight/pom.xml builds the common distribution = managed distribution + SM projects
22:14:30 <abelur> LuisGomez: thanks for confirming
22:14:31 <LuisGomez> in oxygen and before there is no such common distribution
22:20:42 <abelur> LuisGomez: so this change should be only applicable for non-Oxygen AR builds?
22:20:43 <abelur> https://git.opendaylight.org/gerrit/#/c/78224/
22:22:00 <abelur> LuisGomez: so from fluorine onwards??
22:28:48 <LuisGomez> abelur, yes from florine, you can modify above patch command to just ignore failure
22:29:01 <LuisGomez> in case the folder is not there
22:29:12 <abelur> LuisGomez: patch incoming
22:35:25 <abelur> LuisGomez: https://git.opendaylight.org/gerrit/78661
22:56:25 <abelur> LuisGomez: done, its updated now
23:02:48 <abelur> jamoluhrsen: dfarrell07: anyone able to merge that please ?
23:03:14 <abelur> we are blocked on the Oxygen SR2 release and version bumping with that merged?
23:06:38 <dfarrell07> abelur: ack
23:07:54 <dfarrell07> abelur: okay, you have a +2 for when it passes verify
23:08:02 <abelur> dfarrell07: thanks :)
02:13:45 <zxiiro> jamoluhrsen: LuisGomez you guys may see a 403 - Forbidden in some jobs failing to upload logs at the end of the job. I screweed up a permission. sorry about that. It should be fixed now.
02:14:13 <zxiiro> jamoluhrsen: LuisGomez it was broken for about 10 minutes before I caught it so hopefully didn't affect too many jobs
03:51:52 <abelur> does anyone know why the version-bump jobs are failing on dluxapps?
03:51:59 <abelur> 00:36:08 [INFO] ODL :: dluxapps :: yangutils-module 0.7.5-SNAPSHOT . SUCCESS [  0.635 s]
03:51:59 <abelur> 00:36:08 [INFO] ODL :: dluxapps :: yangutils-bundle 0.7.5-SNAPSHOT . FAILURE [  2.126 s]
03:51:59 <abelur> 00:36:08 [INFO] ODL :: dluxapps :: odl-dluxapps-yangman 0.7.5-SNAPSHOT SKIPPED
03:53:08 <abelur> zxiiro: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-version-bump-oxygen/12/console.log.gz
03:59:49 <abelur> jamoluhrsen: dfarrell07 ^^
04:01:03 <abelur> Is this a temporary failure?
06:30:10 <zxiiro> abelur: Return code is: 401, ReasonPhrase: Unauthorized.
06:30:24 <zxiiro> opendaylight.snapshot/org/opendaylight/dluxapps/yangutils-bundle/0.7.5-SNAPSHOT/yangutils-bundle-0.7.5-20181212.003608-1.jar
06:31:48 <zxiiro> I don't know why it's unauthorized for that bundle
06:32:03 <abelur> 401 is very strange?
06:48:37 <abelur> zxiiro: I think thoese failure are likely because of this change: https://git.opendaylight.org/gerrit/#/c/78192/
06:51:47 <abelur> zxiiro: are we setting these vars inside of $GLOBAL_SETTINGS_FILE or SETTINGS_FILE
06:52:52 <zxiiro> abelur: i'm not sure how it's related to a macro that just installs Maven?
06:53:49 <zxiiro> abelur: the lf-provide-maven-settings after it is what passes down the settings files to the job
06:54:12 <zxiiro> abelur: that patch just touches the call that installs mvn, the fact that maven successfully runs means the binary was installed and works.
06:55:58 <abelur> hmmm
06:56:39 <abelur> ok, 401 unauthoried clearly means something to do with invalid creds, unless something changed on Nexus repo. I think these projects are all quite old and we've not touched them since. ... the only place we pass the creds to this version bump job is the config-file-provider
06:56:57 <abelur> diff --git a/jjb/autorelease/autorelease-macros.yaml b/jjb/autorelease/autorelease-macros.yaml
06:56:57 <abelur> index b9edf7fe..a119315b 100644
06:56:57 <abelur> --- a/jjb/autorelease/autorelease-macros.yaml
06:56:57 <abelur> +++ b/jjb/autorelease/autorelease-macros.yaml
06:56:57 <abelur> @@ -39,16 +39,6 @@
06:56:57 <abelur> git clean -dff
06:56:57 <abelur> fi
06:56:58 <abelur> 
06:56:58 <abelur> -- builder:
06:56:59 <abelur> -    name: autorelease-cfp
06:56:59 <abelur> -    builders:
06:57:00 <abelur> -      - config-file-provider:
06:57:42 <abelur> s/invalid/missing/  ^^^ PS
06:57:51 <zxiiro> abelur: so why did many modules complete building before until it got to that one?
06:57:59 <zxiiro> abelur: if it's credentials related then wouldn't it fial on the first module it built?
07:00:11 <abelur> zxiiro: yeah, that's true aswell :~(
07:01:49 <zxiiro> abelur: have you checked nexus logs to see if it has more details?
07:01:52 <abelur> zxiiro: the only other thing I can think off is some issue with ldap
07:02:04 <abelur> i'll also check with the IT team
07:03:41 * abelur checks hte logs first
07:16:48 <abelur> zxiiro: sure if we can see anything from the logs?
07:17:08 <abelur> wondering if this was restarted earlier around the time when the job were going through
07:18:03 <zxiiro> abelur: I don't know. It was a suggestion.
07:19:36 <zxiiro> abelur: nexus was restarted this morning yes but it happened before this job ran
07:20:02 <zxiiro> build #13 https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-version-bump-oxygen/13/
07:20:09 <zxiiro> which failed in the same way
07:20:32 <abelur> zxiiro: that was the second time I ran the job
07:20:39 <abelur> this was the first time: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-version-bump-oxygen/12/console.log.gz
07:21:11 <zxiiro> abelur: considering they both failed with the exact same error I would say the restart was not a factor in this
07:21:22 <abelur> but we hitting the same error both the times ... is not related to the restart
07:21:24 <abelur> yeah
07:23:57 <abelur> zxiiro: looks like the artifacts should have the prefix -SNAPSHOT, which is missing
07:24:06 <abelur> 00:36:08 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project yangutils-bundle: ArtifactDeployerException: Failed to deploy artifacts: Could not transfer artifact org.opendaylight.dluxapps:yangutils-bundle:jar:0.7.5-20181212.003608-1 from/to opendaylight-snapshot::default (https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot): Failed to
07:24:06 <abelur> transfer file: https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/dluxapps/yangutils-bundle/0.7.5-SNAPSHOT/yangutils-bundle-0.7.5-20181212.003608-1.jar. Return code is: 401, ReasonPhrase: Unauthorized. -> [Help 1]
07:24:49 <abelur> so that probably the reason the its returning a 401 unauthorized ... since its a snapshot repo
07:49:21 <zxiiro> abelur: looks like both sigmatopology-bundle and yangutils-bundle both fail to deploy. I can reproduce it
07:49:23 <zxiiro> not sure why though
07:49:56 <zxiiro> the sigmatopology-module and yangutil-module in the same directory deploys successfully no problem though
07:50:11 <zxiiro> very strange...
08:07:39 <zxiiro> abelur: I'd do a manual version bump at this point. dluxapps isn't even a maintained project anymore.
08:40:44 <abelur> zxiiro: ack
16:31:47 <jamoluhrsen> abelur, zxiiro, gerrit question if either of you is around?
16:48:28 <zxiiro> jamoluhrsen: what's the question?
16:48:43 <jamoluhrsen> zxiiro: I am on a review that's already been merged.
16:48:54 <jamoluhrsen> zxiiro: I can't figure out how to remove myself now? not possible
16:48:56 <jamoluhrsen> ?
16:50:17 <zxiiro> jamoluhrsen: yeah seems like it.
16:50:32 <jamoluhrsen> zxiiro: ack. no biggie just thought I was missing something.
16:50:58 <zxiiro> Might want to raise that to the Gerrit folks. Maybe it's intentional.
21:28:44 <abelur> zxiiro: jamoluhrsen trying to merge all these version bump changes ...
21:29:34 <abelur> zxiiro: jamoluhrsen: we have to do this in order ... however the verify is not going through at all
21:29:39 <abelur> for controller
21:29:40 <abelur> https://git.opendaylight.org/gerrit/#/c/78686/
21:30:21 <abelur> *maven-verify* and *validate-autorelease* is not going through?
21:36:53 <jamoluhrsen> abelur: not sure on this, tbh. is it because of snapshot versions for mdsal? don't we just want the release version?
21:37:48 <abelur> nah, the release versions are already promoted :-)
21:38:11 <abelur> this is version bump assuming there is going to be a next release
21:38:43 <jamoluhrsen> abelur: well, you know will know better, but I thought we only consumed released versions of release integrated projects like mdsal.
21:38:46 <abelur> jamoluhrsen: generally we use the version bump job to deploy the snapshots into the Nexus
21:38:49 <jamoluhrsen> anyway, I saw this line in the consolelog:
21:38:50 <jamoluhrsen> 05:39:27 [ERROR] Non-resolvable import POM: Could not find artifact org.opendaylight.mdsal:mdsal-artifacts:pom:2.4.5-SNAPSHOT in opendaylight-snapshot (https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/) @ org.opendaylight.mdsal:binding-parent:0.12.5-SNAPSHOT, /tmp/r/org/opendaylight/mdsal/binding-parent/0.12.5-SNAPSHOT/binding-parent-0.12.5-SNAPSHOT.pom, line 53, column 25
21:41:26 <abelur> hmmm ... that sounds right, let me check with Thanh or Luis once
21:41:54 <abelur> zxiiro: LuisGomez: dfarrell07 anyone has any ideas please ?
21:42:49 * dfarrell07 is trying to figure out the question
21:45:18 <dfarrell07> I think we do always integrate with release versions of MRI projects, if that's the question
21:45:39 <dfarrell07> like MSI project builds should be referencing MRI release artifacts
21:53:45 <zxiiro> abelur: it's not gonna pass until mdsal is merged first.
21:54:04 <zxiiro> dfarrell07: mdsal is not mri in oxygen
21:54:52 <abelur> zxiiro: that's what I was missing ... dint even look for mdsal as its to be top in chart/order
21:55:51 <abelur> thanks :-)
21:56:12 <zxiiro> Alright It's 5 am and the sun is about to rise. Time for bed.
21:56:34 <zxiiro> 6 am rather
22:11:03 <dfarrell07> zxiiro: yikes man, yeah sleepy time
22:13:23 <zxiiro> dfarrell07: the downside of working on the otherside of the world but still have meetings to attend in the US
09:50:58 <abelur> dfarrell07: jamoluhrsen: can I get a merge on these changes please
09:51:04 <abelur> https://git.opendaylight.org/gerrit/78732
09:51:05 <abelur> https://git.opendaylight.org/gerrit/78731
09:51:05 <abelur> https://git.opendaylight.org/gerrit/#/c/78679/
10:11:28 <zxiiro> abelur: merged
10:12:00 <abelur> Zxirro thanks
10:12:11 <zxiiro> abelur: can you cherry-pick to stable/oxygen too the downloads.rst changes?
10:12:24 <abelur> Ok
10:12:37 <abelur> From the master?
10:12:48 <zxiiro> doesn't matter which one you cherry-pick from
10:13:02 <abelur> I think that file is not in stable/oxygen
10:13:12 <abelur> And was brought in later?
10:13:13 <zxiiro> abelur: ok nevermind then
10:13:26 <zxiiro> I don't remember
14:48:19 <zxiiro> jamoluhrsen: dfarrell07 FYI I think I'm going to pass on todays integration meeting. I have too many meetings tonight and would like to get at least some sleep in between.
14:48:45 <dfarrell07> zxiiro: yeah, please do, sleep > meeting
14:48:46 <zxiiro> jamoluhrsen: dfarrell07 updates from me are 1) I think we hopefully shouldn't see gateway timeouts anymore (RELENG-1215) but please keep reporting them if anyone's still seeing them
14:49:35 <zxiiro> jamoluhrsen: dfarrell07 2) Please continue to test https://git.opendaylight.org/gerrit/#/q/topic:global-jjb-stack and give me any feedback. It does require the latest JJB 2.9.2 to work.
14:49:56 <zxiiro> We'll look into rolling it out in the new year
14:52:34 <zxiiro> In other news I'll be back in Ottawa next week (Wednesday) so expect me to be back in usual timezone soon :)
14:55:18 <dfarrell07> zxiiro: coool, thanks for all the good work and have fun :)
22:01:58 <odl_meetbot> jamoluhrsen: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
22:02:05 <jamoluhrsen> #endmeeting