Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on prd-centos8-builder-1c-1g-18135 (centos8-builder-1c-1g) in workspace /w/workspace/ci-management-openstack-cron
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-VmR6HZZlDJrW/agent.5137
SSH_AGENT_PID=5138
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: /w/workspace/ci-management-openstack-cron@tmp/private_key_17545179433161266221.key (jenkins-ssh)
[ssh-agent] Using credentials jenkins (jenkins)
The recommended git tool is: NONE
using credential jenkins
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository ssh://oran-jobbuilder@gerrit.o-ran-sc.org:29418/ci-management
> git init /w/workspace/ci-management-openstack-cron # timeout=10
Fetching upstream changes from ssh://oran-jobbuilder@gerrit.o-ran-sc.org:29418/ci-management
> git --version # timeout=10
> git --version # 'git version 2.39.3'
using GIT_SSH to set credentials jenkins
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
> /usr/bin/chcon --type=ssh_home_t /w/workspace/ci-management-openstack-cron@tmp/jenkins-gitclient-ssh1339873749719382934.key
> git fetch --tags --force --progress -- ssh://oran-jobbuilder@gerrit.o-ran-sc.org:29418/ci-management +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url ssh://oran-jobbuilder@gerrit.o-ran-sc.org:29418/ci-management # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision c27d74fc57b8d3f8acf91bffd89a3b8024497df2 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f c27d74fc57b8d3f8acf91bffd89a3b8024497df2 # timeout=10
Commit message: "Chore: Update global-jjb"
> git rev-list --no-walk c27d74fc57b8d3f8acf91bffd89a3b8024497df2 # timeout=10
> git remote # timeout=10
> git submodule init # timeout=10
> git submodule sync # timeout=10
> git config --get remote.origin.url # timeout=10
> git submodule init # timeout=10
> git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
> git config --get submodule.packer/common-packer.url # timeout=10
> git config -f .gitmodules --get submodule.packer/common-packer.path # timeout=10
> git config --get submodule.global-jjb.url # timeout=10
> git config -f .gitmodules --get submodule.global-jjb.path # timeout=10
> git submodule update --init --recursive packer/common-packer # timeout=10
> git submodule update --init --recursive global-jjb # timeout=10
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [Execute shell]
Run condition [Boolean condition] enabling prebuild for step [BuilderChain]
provisioning config files...
copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
copy managed file [npmrc] to file:/home/jenkins/.npmrc
copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins14614426782649221419.sh
---> python-tools-install.sh
Setup pyenv:
system
* 3.8.13 (set by /opt/pyenv/version)
* 3.9.13 (set by /opt/pyenv/version)
* 3.10.6 (set by /opt/pyenv/version)
lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-2yU8
lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
Generating Requirements File
Python 3.10.6
pip 23.3.1 from /tmp/venv-2yU8/lib/python3.10/site-packages/pip (python 3.10)
appdirs==1.4.4
argcomplete==3.1.6
aspy.yaml==1.3.0
attrs==23.1.0
autopage==0.5.2
Babel==2.13.1
beautifulsoup4==4.12.2
boto3==1.29.0
botocore==1.32.0
bs4==0.0.1
cachetools==5.3.2
certifi==2023.7.22
cffi==1.16.0
cfgv==3.4.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
cliff==4.4.0
cmd2==2.4.3
cryptography==3.3.2
debtcollector==2.5.0
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.14
distlib==0.3.7
dnspython==2.4.2
docker==4.2.2
dogpile.cache==1.2.2
email-validator==2.1.0.post1
filelock==3.13.1
future==0.18.3
gitdb==4.0.11
GitPython==3.1.40
google-auth==2.23.4
httplib2==0.22.0
identify==2.5.31
idna==3.4
importlib-metadata==6.8.0
importlib-resources==1.5.0
iso8601==2.1.0
Jinja2==3.1.2
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.19.2
jsonschema-specifications==2023.11.1
keystoneauth1==5.3.0
kubernetes==28.1.0
lftools==0.37.8
lxml==4.9.3
MarkupSafe==2.1.3
msgpack==1.0.7
multi-key-dict==2.0.3
munch==4.0.0
netaddr==0.9.0
netifaces==0.11.0
niet==1.4.2
nodeenv==1.8.0
oauth2client==4.1.3
oauthlib==3.2.2
openstacksdk==0.62.0
os-client-config==2.1.0
os-service-types==1.7.0
osc-lib==2.9.0
oslo.config==9.2.0
oslo.context==5.3.0
oslo.i18n==6.2.0
oslo.log==5.4.0
oslo.serialization==5.2.0
oslo.utils==6.3.0
packaging==23.2
pbr==6.0.0
platformdirs==3.11.0
prettytable==3.9.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pygerrit2==2.0.15
PyGithub==2.1.1
pyinotify==0.9.6
PyJWT==2.8.0
PyNaCl==1.5.0
pyparsing==2.4.7
pyperclip==1.8.2
pyrsistent==0.20.0
python-cinderclient==9.4.0
python-dateutil==2.8.2
python-heatclient==3.3.0
python-jenkins==1.8.2
python-keystoneclient==5.2.0
python-magnumclient==4.2.0
python-novaclient==18.4.0
python-openstackclient==6.0.0
python-swiftclient==4.4.0
pytz==2023.3.post1
PyYAML==6.0.1
referencing==0.31.0
requests==2.31.0
requests-oauthlib==1.3.1
requestsexceptions==1.4.0
rfc3986==2.0.0
rpds-py==0.12.0
rsa==4.9
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
s3transfer==0.7.0
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
soupsieve==2.5
stevedore==5.1.0
tabulate==0.9.0
toml==0.10.2
tomlkit==0.12.3
tqdm==4.66.1
typing_extensions==4.8.0
tzdata==2023.3
urllib3==1.26.18
virtualenv==20.24.6
wcwidth==0.2.10
websocket-client==1.6.4
wrapt==1.16.0
xdg==6.0.0
xmltodict==0.13.0
yq==3.2.3
zipp==3.17.0
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
OS_CLOUD=vex
[EnvInject] - Variables injected successfully.
[Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
Run condition [Boolean condition] enabling perform for step [Execute shell]
[ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins17714972832117308647.sh
---> Orphaned k8s clusters
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: kubernetes python-heatclient python-openstackclient python-magnumclient
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lftools 0.37.8 requires openstacksdk<1.5.0, but you have openstacksdk 2.0.0 which is incompatible.
virtualenv 20.24.6 requires platformdirs<4,>=3.9.1, but you have platformdirs 4.0.0 which is incompatible.
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
-----> Delete orphaned cluster
[Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
Run condition [Boolean condition] enabling perform for step [Execute shell]
[ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins7730121531588913352.sh
---> Orphaned stacks
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
+ mapfile -t OS_COE_CLUSTERS_ID
++ openstack --os-cloud vex coe cluster list -f value -c uuid -c name
++ grep -E '(DELETE_FAILED|UNKNOWN|UNHEALTHY)'
++ awk '{print $1}'
+ echo '-----> Active clusters -> stacks'
-----> Active clusters -> stacks
+ OS_COE_STACKS=()
+ [[ 0 -gt 0 ]]
+ mapfile -t OS_STACKS
++ openstack --os-cloud vex stack list -f value -c 'Stack Name' -c 'Stack Status' --property stack_status=CREATE_COMPLETE --property stack_status=DELETE_FAILED --property stack_status=CREATE_FAILED
++ awk '{print $1}'
+ echo '-----> Active stacks'
-----> Active stacks
+ echo '-----> Delete orphaned stacks'
-----> Delete orphaned stacks
[Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
Run condition [Boolean condition] enabling perform for step [Execute shell]
[ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins4058898102719475264.sh
---> Orphaned servers
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
-----> Active servers
prd-centos8-builder-1c-1g-18135
-----> Delete orphaned servers
ERROR: Failed to fetch data from https://jenkins.o-ran-sc.org/sandbox/computer/api/json?tree=computer[displayName] with status code 502
502 Bad Gateway
502 Bad Gateway
nginx/1.24.0
502
Build step 'Conditional step (single)' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 5138 killed;
[ssh-agent] Stopped.
[PostBuildScript] - [INFO] Executing post build scripts.
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins18088517548701189444.sh
---> sysstat.sh
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins13836759771184917888.sh
---> package-listing.sh
++ facter osfamily
++ tr '[:upper:]' '[:lower:]'
+ OS_FAMILY=redhat
+ workspace=/w/workspace/ci-management-openstack-cron
+ START_PACKAGES=/tmp/packages_start.txt
+ END_PACKAGES=/tmp/packages_end.txt
+ DIFF_PACKAGES=/tmp/packages_diff.txt
+ PACKAGES=/tmp/packages_start.txt
+ '[' /w/workspace/ci-management-openstack-cron ']'
+ PACKAGES=/tmp/packages_end.txt
+ case "${OS_FAMILY}" in
+ rpm -qa
+ sort
+ '[' -f /tmp/packages_start.txt ']'
+ '[' -f /tmp/packages_end.txt ']'
+ diff /tmp/packages_start.txt /tmp/packages_end.txt
+ '[' /w/workspace/ci-management-openstack-cron ']'
+ mkdir -p /w/workspace/ci-management-openstack-cron/archives/
+ cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/ci-management-openstack-cron/archives/
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins3124512449954703854.sh
---> capture-instance-metadata.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
INFO: Running in OpenStack, capturing instance metadata
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins10595597462591333680.sh
provisioning config files...
copy managed file [jenkins-log-archives-settings] to file:/w/workspace/ci-management-openstack-cron@tmp/config262844036314196413tmp
Regular expression run condition: Expression=[^.*logs-s3.*], Label=[]
Run condition [Regular expression match] preventing perform for step [Provide Configuration files]
[EnvInject] - Injecting environment variables from a build step.
[EnvInject] - Injecting as environment variables the properties content
SERVER_ID=logs
[EnvInject] - Variables injected successfully.
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins6826394578466544911.sh
---> create-netrc.sh
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins1198733509997183061.sh
---> python-tools-install.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins18015562706855797089.sh
---> sudo-logs.sh
Archiving 'sudo' log..
[ci-management-openstack-cron] $ /bin/bash /tmp/jenkins7710402527513757159.sh
---> job-cost.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lftools 0.37.8 requires openstacksdk<1.5.0, but you have openstacksdk 2.0.0 which is incompatible.
virtualenv 20.24.6 requires platformdirs<4,>=3.9.1, but you have platformdirs 4.0.0 which is incompatible.
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
INFO: No Stack...
INFO: Retrieving Pricing Info for: v2-highcpu-1
INFO: Archiving Costs
[ci-management-openstack-cron] $ /bin/bash -l /tmp/jenkins4286241027516304116.sh
---> logs-deploy.sh
Setup pyenv:
system
3.8.13
3.9.13
* 3.10.6 (set by /w/workspace/ci-management-openstack-cron/.python-version)
lf-activate-venv(): INFO: Reuse venv:/tmp/venv-2yU8 from file:/tmp/.os_lf_venv
lf-activate-venv(): INFO: Installing: lftools
lf-activate-venv(): INFO: Adding /tmp/venv-2yU8/bin to PATH
INFO: Nexus URL https://nexus.o-ran-sc.org path production/vex-sjc-oran-jenkins-prod-1/ci-management-openstack-cron/39544
INFO: archiving workspace using pattern(s):
Archives upload complete.
INFO: archiving logs to Nexus
---> uname -a:
Linux prd-centos8-builder-1c-1g-18135.novalocal 4.18.0-496.el8.x86_64 #1 SMP Mon Jun 5 15:04:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
---> lscpu:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC-Rome Processor
Stepping: 0
CPU MHz: 2799.998
BogoMIPS: 5599.99
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 16384K
NUMA node0 CPU(s): 0
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
---> nproc:
1
---> df -h:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 330M 0 330M 0% /dev
tmpfs 386M 0 386M 0% /dev/shm
tmpfs 386M 11M 376M 3% /run
tmpfs 386M 0 386M 0% /sys/fs/cgroup
/dev/vda1 20G 8.5G 12G 43% /
tmpfs 78M 0 78M 0% /run/user/1001
---> free -m:
total used free shared buff/cache available
Mem: 770 312 83 5 374 327
Swap: 1023 28 995
---> ip addr:
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:1d:32:dd brd ff:ff:ff:ff:ff:ff
altname enp0s3
altname ens3
inet 10.32.6.190/23 brd 10.32.7.255 scope global dynamic noprefixroute eth0
valid_lft 86125sec preferred_lft 86125sec
inet6 fe80::f816:3eff:fe1d:32dd/64 scope link
valid_lft forever preferred_lft forever
---> sar -b -r -n DEV:
Linux 4.18.0-496.el8.x86_64 (centos-stream-8-builder-648259bc-7a64-b836-c386-c6b0ebb25af2.nov) 11/15/2023 _x86_64_ (1 CPU)
18:03:47 LINUX RESTART (1 CPU)
06:04:01 PM tps rtps wtps bread/s bwrtn/s
06:05:01 PM 252.47 171.80 80.67 12289.28 40918.25
06:06:02 PM 167.73 34.49 133.24 3701.57 17532.38
06:07:01 PM 122.82 38.56 84.26 2216.64 6790.12
06:08:01 PM 140.68 110.66 30.02 7610.86 1109.06
Average: 171.16 89.05 82.11 6471.29 16641.88
06:04:01 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
06:05:01 PM 60040 313188 729016 92.39 2100 331640 517584 28.17 130564 463312 21452
06:06:02 PM 73960 298648 715096 90.63 200 291628 520296 28.31 100984 442520 28132
06:07:01 PM 155524 336492 633532 80.29 28 246456 482040 26.23 121600 338192 12436
06:08:01 PM 85040 345248 704016 89.22 0 325848 476900 25.95 144428 391864 2980
Average: 93641 323394 695415 88.13 582 298893 499205 27.17 124394 408972 16250
06:04:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
06:05:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:05:01 PM eth0 194.96 124.12 978.68 53.67 0.00 0.00 0.00 0.00
06:06:02 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:06:02 PM eth0 58.80 54.68 860.05 6.63 0.00 0.00 0.00 0.00
06:07:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:07:01 PM eth0 21.70 22.87 182.54 7.80 0.00 0.00 0.00 0.00
06:08:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:08:01 PM eth0 17.71 18.49 17.77 11.88 0.00 0.00 0.00 0.00
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 73.55 55.21 511.86 20.05 0.00 0.00 0.00 0.00
---> sar -P ALL:
Linux 4.18.0-496.el8.x86_64 (centos-stream-8-builder-648259bc-7a64-b836-c386-c6b0ebb25af2.nov) 11/15/2023 _x86_64_ (1 CPU)
18:03:47 LINUX RESTART (1 CPU)
06:04:01 PM CPU %user %nice %system %iowait %steal %idle
06:05:01 PM all 61.65 0.10 9.22 17.28 0.12 11.64
06:05:01 PM 0 61.65 0.10 9.22 17.28 0.12 11.64
06:06:02 PM all 70.17 0.00 6.87 5.61 0.12 17.23
06:06:02 PM 0 70.17 0.00 6.87 5.61 0.12 17.23
06:07:01 PM all 60.47 0.00 4.67 6.92 0.10 27.83
06:07:01 PM 0 60.47 0.00 4.67 6.92 0.10 27.83
06:08:01 PM all 57.95 0.00 3.14 15.06 0.10 23.75
06:08:01 PM 0 57.95 0.00 3.14 15.06 0.10 23.75
Average: all 62.58 0.03 5.98 11.23 0.11 20.07
Average: 0 62.58 0.03 5.98 11.23 0.11 20.07