18:16:39 <mattw4> #startmeeting cperf
18:16:39 <collabot> Meeting started Thu Aug 24 18:16:39 2017 UTC.  The chair is mattw4. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:16:39 <collabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:16:39 <collabot> The meeting name has been set to 'cperf'
18:18:48 <mattw4> #info jamoluhrsen got upstream tweaks to run robot scale tests in cperf pod
18:20:08 <mattw4> #info sai started latest round of scale testing
18:20:31 <mattw4> #info env is running with clustered ODL
18:21:11 <mattw4> #info started with 3 controllers & 3 computes
18:21:18 <dfarrell07> #link https://nexus.opendaylight.org/content/repositories/opendaylight-carbon-epel-7-x86_64-devel/org/opendaylight/integration-packaging/opendaylight/6.2.0-0.1.20170824rel1941.el7.noarch/opendaylight-6.2.0-0.1.20170824rel1941.el7.noarch.rpm Sai may want to consider using this RPM, it has no-IP fix Tim wanted for OPNFV
18:21:42 <mattw4> #info much harder to hit OOM this time (with ODL defaults)
18:22:45 <mattw4> #info encountering bugs similar to memory leaks & Java failures seen previously
18:22:51 <sai> https://bugs.opendaylight.org/show_bug.cgi?id=9034
18:23:02 <mattw4> #info ODL version == prerelease SR2 build
18:23:31 <sai> dfarrell07: opendaylight-6.2.0-0.1.20170817rel1931.el7.noarch
18:23:42 <mattw4> #info ODL should have memory leak fix, but still may fail clustering
18:24:39 <mattw4> #info we likely have more memory leaks to fix
18:25:43 <mattw4> #info second issue under investigation is networking-odl driver itself
18:25:55 <mattw4> #info journal driver getting hit hard
18:26:23 <mattw4> #info workaround is to reduce number of worker processes
18:27:04 <mattw4> #info mysql has several queries that are taking a long time and consuming a lot of resources
18:28:03 <mattw4> #info browbeat tool helping to monitor resources, but missing java info - sai has added a plugin
18:28:46 <mattw4> #info need deep-dive into java resource consumption to understand better
18:29:13 <sai> https://snapshot.raintank.io/dashboard/snapshot/YGlm4u58NFJzMzgNhLYDiYGHX2pFKUdy?orgId=2&panelId=226&fullscreen&from=1503579973482&to=1503583573482&var-Cloud=odl-osp12-non-containerized&var-NodeType=*&var-Node=overcloud-controller-2&var-Interface=interface-br-ex&var-Disk=disk-sda&var-OSD=All&var-Pool=All&var-cpus0=All&var-cpus00=All
18:29:33 <sai> https://snapshot.raintank.io/dashboard/snapshot/emsmNggId7uHKq9S1r2xX5qhBew2GCP6?orgId=2&panelId=69&fullscreen&from=1503595117077&to=1503598717077&var-Cloud=odl-osp12-non-containerized&var-NodeType=*&var-Node=overcloud-controller-0&var-Interface=interface-br-ex&var-Disk=disk-sda&var-OSD=All&var-Pool=All&var-cpus0=All&var-cpus00=All
18:30:41 <mattw4> #info sai shared lovely grafana dashboards showing resource consumption
18:33:47 <mattw4> #info sai still seeing lots of slow queries to mysql
18:34:25 <mattw4> #info mattw4 to take AR to follow up with LF
18:35:24 <mattw4> #info mattw4 needs to learn more about robot
18:37:38 <mattw4> #info mattw4 to ping sai for help on monitoring
18:37:57 <mattw4> #endmeeting