13:56:44 <mackonstan> #startmeeting FD.io CSIT Project Meeting
13:56:44 <collab-meetbot`> Meeting started Wed Jul 21 13:56:44 2021 UTC and is due to finish in 60 minutes.  The chair is mackonstan. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:56:44 <collab-meetbot`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:56:44 <collab-meetbot`> The meeting name has been set to 'fd_io_csit_project_meeting'
13:58:25 <pmikus> #info Peter Mikus
14:01:15 <mackonstan> #info mackonstan
14:01:17 <mackonstan> #chair
14:01:17 <collab-meetbot`> Current chairs: mackonstan
14:01:57 <vrpolak> #info Vratko Polak.
14:02:06 <dwallacelf> #info Dave Wallace
14:02:53 <mackonstan> #topic Agenda bashing
14:03:43 <mackonstan> #topic Inputs from projects - VPP
14:04:07 <mackonstan> #topic Inputs from projects - TSC
14:04:09 <jlinkes> #info Juraj LinkeŇ°
14:05:42 <mackonstan> #info Dave, Peter Mikus nominated as ci-management committer, all committers voted in favour, TSC approval this week.
14:06:09 <mackonstan> #topic New tests
14:06:51 <fan_zhang> #info Fan Zhang
14:08:08 <mackonstan> #info Fan, re flow "offload" patch functional test, waiting for vpp_device testbed/framework update.
14:09:20 <mackonstan> #info Fan, re flow "offload" patch performance test, testbeds/framework ready, will ask Yulong to go ahead and create a patch for that.
14:11:42 <mackonstan> #info Maciek, re TRex support for DPDK CVL, will send a summary note re status, will need help to verify any remaining gaps.
14:13:41 <mackonstan> #info Juraj, re IPsec SPD flow cache patches, 3 patches open, found one issues, working on a fix. Vratko wants to run perf tests to see any collateral impact of those patches.
14:14:14 <mackonstan> #topic Releases
14:15:27 <mackonstan> #info CSIT-2110, will follow VPP schedule
14:15:29 <mackonstan> #link https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_21.10
14:17:30 <mackonstan> #info CSIT-2106, Tibor, still testing (delayed by jenkins issues, talked about later), 2n-zn2 1 cvg job, af_xdp driver tests (late arrival to job_specs).
14:19:54 <mackonstan> #info CSIT-2106, Tibor, methodology sections under review, ipsec async patch, uni-dir crypto, af_xdp content to be added (Peter, Maciek).
14:21:51 <mackonstan> #info CSIT-2106, Peter, MLX issue fixed for good.
14:23:22 <mackonstan> #info CSIT-2106, Vratko, re IPsec async broken test, still open; Fan, vpp patch may need to be re-worked.
14:25:01 <mackonstan> #info CSIT-2106, high latency issue reported by TRex, Vratko, one very clear symptom: latency running high to 5msec, some NICs affected some are not, same for testbeds.
14:25:51 <mackonstan> #info Vratko, issue appeared when testbeds got upgraded to Ubuntu 20.04 and TRex v2.88. Unclear what is the root cause. One similar symptom reported in public domain for TRex.
14:30:50 <vrpolak> #link https://github.com/cisco-system-traffic-generator/trex-core/issues/508#issuecomment-662363667  <-- Known TRex issue, comment about impact on latency measurements, but for different firmware and TRex versions.
14:31:43 <mackonstan> #info Peter, also tried TRex v2.90, also observing high latency.
14:34:20 <mackonstan> #info Peter, another suspicion is PCIe bus management, checking ...
14:37:35 <mackonstan> #info CSIT-2101, vpp artefacts ready, will test once CSIT-2106 is complete, after latency issue is fixed, and one iterative job per testbed is run with latency fixed. Then CSIT-2101.1 maintenance rls tests will get scheduled.
14:37:53 <mackonstan> #topic Physical Infrastructure
14:38:36 <mackonstan> #info Peter, going ahead with 3n-hsw decommissioning, tickets open. Waiting for Vexxhost engineer on-site engineer allocation.
14:38:51 <mackonstan> #topic Virtual Infrastructure, CI/CD
14:39:31 <mackonstan> Dave, Maciek, escalated Jenkins outage issue
14:39:44 <mackonstan> #info Dave, Maciek, escalated Jenkins outage issue
14:44:14 <mackonstan> #info Dave, trigger: last Wednesday Jenkins VM and Jenkins ingress got moved to different location(s), some other changes on Saturday; symptoms: as of Saturday, very high latency between jenkins ingress and nomad; resolution: migrated jenkins ingress somewhere else; Peter cleaned vpp_device jobs, Dave cleared the vpp job queue. Now back to normal
14:44:14 <mackonstan> operation.
14:46:22 <mackonstan> #info Dave, jenkins maintenance not needed
14:47:11 <mackonstan> #info Dave, Vanessa applied Jenkins JCasC changes during the outage.
14:47:20 <mackonstan> #info Peter, Dave, Maciek
14:47:53 <mackonstan> #info Peter, Dave, Maciek, changes of this nature MUST BE communicated over the slack channel.
14:51:04 <mackonstan> #info Dave, Maciek: will request post mortem readout from Vexxhost/LFIT team.
14:51:54 <mackonstan> #topic Trending
14:52:24 <mackonstan> #info Vratko, CSIT l2, ip4, ip6 scale tests are failing due to VPP removing support for VAT APIs used.
14:53:58 <mackonstan> #info Vratko, solution is still WIP.
14:54:59 <mackonstan> #endmeeting