=================== #opnfv-cperf: cperf =================== Meeting started by mattw4 at 18:16:39 UTC. The full logs are available at http://ircbot.wl.linuxfoundation.org/meetings/opnfv-cperf/2017/opnfv-cperf.2017-08-24-18.16.log.html . Meeting summary --------------- * jamoluhrsen got upstream tweaks to run robot scale tests in cperf pod (mattw4, 18:18:48) * sai started latest round of scale testing (mattw4, 18:20:08) * env is running with clustered ODL (mattw4, 18:20:31) * started with 3 controllers & 3 computes (mattw4, 18:21:11) * LINK: https://nexus.opendaylight.org/content/repositories/opendaylight-carbon-epel-7-x86_64-devel/org/opendaylight/integration-packaging/opendaylight/6.2.0-0.1.20170824rel1941.el7.noarch/opendaylight-6.2.0-0.1.20170824rel1941.el7.noarch.rpm Sai may want to consider using this RPM, it has no-IP fix Tim wanted for OPNFV (dfarrell07, 18:21:18) * much harder to hit OOM this time (with ODL defaults) (mattw4, 18:21:42) * encountering bugs similar to memory leaks & Java failures seen previously (mattw4, 18:22:45) * LINK: https://bugs.opendaylight.org/show_bug.cgi?id=9034 (sai, 18:22:51) * ODL version == prerelease SR2 build (mattw4, 18:23:02) * ODL should have memory leak fix, but still may fail clustering (mattw4, 18:23:42) * we likely have more memory leaks to fix (mattw4, 18:24:39) * second issue under investigation is networking-odl driver itself (mattw4, 18:25:43) * journal driver getting hit hard (mattw4, 18:25:55) * workaround is to reduce number of worker processes (mattw4, 18:26:23) * mysql has several queries that are taking a long time and consuming a lot of resources (mattw4, 18:27:04) * browbeat tool helping to monitor resources, but missing java info - sai has added a plugin (mattw4, 18:28:03) * need deep-dive into java resource consumption to understand better (mattw4, 18:28:46) * LINK: https://snapshot.raintank.io/dashboard/snapshot/YGlm4u58NFJzMzgNhLYDiYGHX2pFKUdy?orgId=2&panelId=226&fullscreen&from=1503579973482&to=1503583573482&var-Cloud=odl-osp12-non-containerized&var-NodeType=*&var-Node=overcloud-controller-2&var-Interface=interface-br-ex&var-Disk=disk-sda&var-OSD=All&var-Pool=All&var-cpus0=All&var-cpus00=All (sai, 18:29:13) * LINK: https://snapshot.raintank.io/dashboard/snapshot/emsmNggId7uHKq9S1r2xX5qhBew2GCP6?orgId=2&panelId=69&fullscreen&from=1503595117077&to=1503598717077&var-Cloud=odl-osp12-non-containerized&var-NodeType=*&var-Node=overcloud-controller-0&var-Interface=interface-br-ex&var-Disk=disk-sda&var-OSD=All&var-Pool=All&var-cpus0=All&var-cpus00=All (sai, 18:29:33) * sai shared lovely grafana dashboards showing resource consumption (mattw4, 18:30:41) * sai still seeing lots of slow queries to mysql (mattw4, 18:33:47) * mattw4 to take AR to follow up with LF (mattw4, 18:34:25) * mattw4 needs to learn more about robot (mattw4, 18:35:24) * mattw4 to ping sai for help on monitoring (mattw4, 18:37:38) Meeting ended at 18:37:57 UTC. People present (lines said) --------------------------- * mattw4 (22) * sai (4) * collabot (3) * dfarrell07 (1) Generated by `MeetBot`_ 0.1.4