Unable to create OVS interface; Error: ovsdb: Could not connect: Permission denied
Hi,
I deployed an Openshift 4.7.31 IPI cluster but I am unable to add OVS interface on the OVS uplink port using nmcli.
I followed the instructions at https://developer-old.gnome.org/NetworkManager/stable/nm-openvswitch.html to create the connection profiles and OVS bridge, ports, and interfaces as shown below:
[root@ipicluster1-bmdjn-master-1 ~]# nmcli c
NAME UUID TYPE DEVICE
Wired connection 2 63e37c5b-c364-39f7-bea7-79d383cfa711 ethernet ens224
Wired connection 1 505b3190-9b6d-369f-91b2-ac0e55061592 ethernet ens192
br-int 43e8afe0-f3c1-4f7d-a57c-674e3cfaedcd ovs-bridge --
br-int-ovs-intf 831ba8b8-daa3-4eee-8d5e-d02283e5e7eb ovs-interface --
br-int-ovs-port 03e3611b-8406-4477-b857-6274b1bd3912 ovs-port --
ens192-ovs-intf 33ac107f-8b00-49c4-a4ea-2355ab3a311a ethernet -- <-- This cannot be activated
ens192-ovs-port 5192933f-bd2e-4ada-9351-a6d07d02b0a7 ovs-port --
I started OVS daemons:
[root@ipicluster1-bmdjn-master-1 ~]# /usr/share/openvswitch/scripts/ovs-ctl start
/etc/openvswitch/conf.db does not exist ... (warning).
Creating empty database /etc/openvswitch/conf.db.
Starting ovsdb-server.
system ID not configured, please use --system-id ... failed!
Configuring Open vSwitch system IDs.
Starting ovs-vswitchd.
Enabling remote OVSDB managers.
[root@ipicluster1-bmdjn-master-1 ~]# /usr/share/openvswitch/scripts/ovs-ctl status
ovsdb-server is running with pid 259485
ovs-vswitchd is running with pid 259498
and manually tried to create the OVS bridge using nmcli, but it failed:
[root@ipicluster1-bmdjn-master-1 ~]# nmcli c down 505b3190-9b6d-369f-91b2-ac0e55061592 && nmcli c up 43e8afe0-f3c1-4f7d-a57c-674e3cfaedcd && nmcli c up 03e3611b-8406-4477-b857-6274b1bd3912 && nmcli c up 5192933f-bd2e-4ada-9351-a6d07d02b0a7 && nmcli c up 33ac107f-8b00-49c4-a4ea-2355ab3a311a && nmcli c up 831ba8b8-daa3-4eee-8d5e-d02283e5e7eb
Connection 'Wired connection 1' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/356)
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/357)
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/358)
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/359)
Error: Connection activation failed: Open vSwitch database connection failed
Hint: use 'journalctl -xe NM_CONNECTION=33ac107f-8b00-49c4-a4ea-2355ab3a311a + NM_DEVICE=ens192' to get more details.
[root@ipicluster1-bmdjn-master-1 ~]# nmcli c
NAME UUID TYPE DEVICE
Wired connection 2 63e37c5b-c364-39f7-bea7-79d383cfa711 ethernet ens224
br-int 43e8afe0-f3c1-4f7d-a57c-674e3cfaedcd ovs-bridge br-int
br-int-ovs-port 03e3611b-8406-4477-b857-6274b1bd3912 ovs-port br-int
ens192-ovs-port 5192933f-bd2e-4ada-9351-a6d07d02b0a7 ovs-port ens192-ovs-port
br-int-ovs-intf 831ba8b8-daa3-4eee-8d5e-d02283e5e7eb ovs-interface --
ens192-ovs-intf 33ac107f-8b00-49c4-a4ea-2355ab3a311a ethernet --
Wired connection 1 505b3190-9b6d-369f-91b2-ac0e55061592 ethernet --
journalctl showing error:
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.8988] agent-manager: agent[c687162a38fa692f,:1.3327/nmcli-connect/0]: agent registered
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9044] device (ens192): Activation: starting connection 'ens192-ovs-intf' (33ac107f-8b00-49c4-a4ea-2355ab3a311a)
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9047] audit: op="connection-activate" uuid="33ac107f-8b00-49c4-a4ea-2355ab3a311a" name="ens192-ovs-intf" pid=256103 uid=0 result="success"
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9048] device (ens192): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9063] device (ens192): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9098] device (ens192): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9105] device (ens192): Activation: connection 'ens192-ovs-intf' enslaved, continuing activation
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9135] device (ens192): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9147] ovsdb: Could not connect: Permission denied
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <warn> [1633557005.9147] device ens192 could not be added to a ovs port: disconnected from ovsdb
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9147] device (ens192): state change: ip-check -> failed (reason 'ovsdb-failed', sys-iface-state: 'managed')
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9152] device (ens192-ovs-port): releasing ovs interface ens192
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9152] device (ens192): released from master device ens192-ovs-port
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <warn> [1633557005.9165] device (ens192): Activation: failed for connection 'ens192-ovs-intf'
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <warn> [1633557005.9168] device ens192 could not be removed from a ovs port: disconnected from ovsdb
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9168] ovsdb: disconnected from ovsdb
Oct 06 21:50:05 ipicluster1-bmdjn-master-1 NetworkManager[1073]: <info> [1633557005.9172] device (ens192): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed’)
It seems that NetworkManager does not have Permission to connect to db.sock.
Some useful info:
[root@ipicluster1-bmdjn-master-1 ~]# ls /var/run/openvswitch/
db.sock ovsdb-server.259485.ctl ovsdb-server.pid ovs-vswitchd.259498.ctl ovs-vswitchd.pid
[root@ipicluster1-bmdjn-master-1 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="47.84.202109092331-0"
VERSION_ID="4.7"
OPENSHIFT_VERSION="4.7"
RHEL_VERSION="8.4"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 47.84.202109092331-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.7"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.7"
OSTREE_VERSION='47.84.202109092331-0'
[root@ipicluster1-bmdjn-master-1 ~]# uname -r
4.18.0-305.19.1.el8_4.x86_64
[root@ipicluster1-bmdjn-master-1 ~]# systemctl status NetworkManager
● NetworkManager.service - Network Manager
Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/NetworkManager.service.d
└─NetworkManager-ovs.conf
Active: active (running) since Wed 2021-10-06 19:22:17 UTC; 2h 46min ago
Docs: man:NetworkManager(8)
Main PID: 1073 (NetworkManager)
Tasks: 3 (limit: 153571)
Memory: 9.0M
CPU: 13.186s
CGroup: /system.slice/NetworkManager.service
└─1073 /usr/sbin/NetworkManager --no-daemon
[root@ipicluster1-bmdjn-master-1 ~]# nmcli -version
nmcli tool, version 1.30.0-10.el8_4
Is there a step that I might be missing or having some wrong configuration?
The same configuration (nm connection profiles) used to work before with NMCLI version:
[core@ipicluster1-ttjnc-master-1 ~]$ nmcli -version
nmcli tool, version 1.26.0-14.1.rhaos4.7.el8